Scale Mesa to Model Millions of Agents

[Srj]

Is there a way i can take advantages of my multiple core to parallelize computation of mesa. For example, In Wolf-Sheep predator model if i use 100k wolf and 300k Sheep the model runs very very slowly. I guess there is a way to parallalize the operation?

[tpike3]

I have not found a good way to parallalize a single model. As Python can really only do multiprocessing due to the Global interpreter lock (GIL) I tried using message passing to split a model spatially and pass as agent crossed the boundaries. But in general message passing is too slow so yo don't gain a lot of efficiencies.

I think the more appropriate way would be to convert a model to arrays but I have not attempted that yet. Defnieitely interested in more discussion on the topic.

[Srj]

How about using concurrent Process pool executor? https://docs.python.org/3/library/concurrent.futures.html

[tpike3]

@Srj I personally have not tried that. But, if you can find a way I am all for it.

The challenge we keep running into is how to do it so it is generic for any model vs the optimization of one model.

[rht]

concurrent.futures is just another different API to schedule executions, and so using it won't necessarily result in a faster code than using multiprocessing.

[jackiekazil]

I am late to the party. This is all very interesting. Has anyone explored any of these?

[rht]

@DavoudTaghawiNejad has recently written a model using Numpy arrays to store agents states.

[TabeaSonnenschein]

Hi everyone, interesting discussion! Any update on an easy implementation of multiprocessing or concurrent.future for the agent's steps in the mesa model? Maybe in 2 years the outlook has changed? Thanks :).

[TabeaSonnenschein]

@Srj , thanks for explaining. Fortunately, the agents do not depend on each other during a step. They just have a collective impact on the environment. Why do you think the parallelization did not improve the speed? Was that the shared resources and overhead issue?

@Srj and @rht, if you have example code on how you implemented the parallelization and/or you represent individual agents as rows in an array or DF, that would help me immensely!

[rht]

Can you post a minimal reproducible demo of your model, with unnecessary details simplified?

[TabeaSonnenschein]

I excluded some major chucks but left a sceleton. Does that work? Without the data, you cannot run it anyway, I guess. Thanks for your time. I truly appreciate! :-)

** The model is still under development

[rht]

Can you remove small unnecessary details. Just keep the 0th or 1st order effect. This is so that it is easy to explore, before applying it on the full version. Have you tried using https://github.com/pyutils/line_profiler to measure which part is the slowest? Also, linking to a file posted on a GitHub repo is better.

[TabeaSonnenschein]

@everyone . Thanks for thinking along with me. It really helped me. Yesterday, I found out that the slowness of my ABM parallelization efforts on Windows was caused by the fact that Windows uses "spawning" to start new workers, which creates new versions of all the variables for each worker. On the other hand Linux allows for a technique called "forking", in which the child process inherits all the same variables in the same state as they were in the parent. Each child process then continues independently from the forking point. This reduces the overhead of dataintensive tasks to the transfer of variables that are actually changing.

Using that new knowledge I set up my WSL Ubuntu and tried out the concurrent.futures (cf) ProcessPoolExecutor. In other words, the good news is that I managed to parallelize the ABM and gain significant speed improvements. Yey! To make things explicit for future computational venturers:

Creating agents during init. Map the Class function (in my case Humans) to the iteratables. Best to load all data that is needed in the init function of the agents before so that it can be accessed with the iterable. You can add as many iterables as you want to the map functions, but try keep it as short as possible.

        with cf.ProcessPoolExecutor() as executor:
            agents = executor.map(Humans, range(self.nb_humans), it.repeat(self, self.nb_humans)) 
        for agent in agents:
            self.continoussp.place_agent(agent, agent.Residence)
            self.schedule.add(agent)

Now to the interesting part: Parallelizing the agent's step function. For that you replace the "self.schedule.step()" of the model.step function with the code below. Interestingly, the ThreadPoolExecutor automatically saved all updated agent variables when simply executing the Humans.step function. However, it does not yet use all CPU power. When exchanging the ThreadPoolExecutor with the ProcessPoolExecutor all CPU power is used, however the variables are not yet saved. I am still figuring out how that works.

        with cf.ThreadPoolExecutor() as executor:
            executor.map(Humans.step, self.schedule.agents, it.repeat(self.current_datetime, self.nb_humans) )    

Anyhow, during my journey (into the unknown), I noticed that you can very easily also parallelize only subfunctions of the agents. This allows bypassing the interaction problem that you all keep mentioning, and which I agree with. E.g. So you can parallelize all functions that are independent of agent interaction with something similar to the code below.

        with cf.ProcessPoolExecutor() as executor:
           for agent in self.schedule.agents:     
               if agent.traveldecision == 1:
                 agent.RouteVars, agent.OrigVars, agent.DestVars = executor.submit(PerceiveEnvironment,agent.route_eucl_line, agent.pos, agent.destination_activity, self.EnvBehavDeterms).result()                

Then you can call functions that are depedent on interaction in the agent.step function using "self.schedule.step()" before or after. Or/And you can also iterate through the agents to call that function before or after the parallelized function call, e.g.

   for agent in self.schedule.agents:     
            agent.YourFunction-()

Hope this helps others! @rht , yes indeed, profilers are useful for optimization too!

[Srj]

Why do you think the parallelization did not improve the speed? Was that the shared resources and overhead issue?

@TabeaSonnenschein I think in my case it didn't improve because each agent had to interact with all other agents. As you said this is not the case for you, it might improve the speed in your case. It would be great if you give an update here when/if you implement it.

[TabeaSonnenschein]

@Srj yes that seems like an exponential problem if they have to interact with ALL other agents. My issue was the resource management, since I have a lot of data in my model and the transfer to all other workers is slow. However, I found a solution based on Linux fork technique: (see response here #1033 (reply in thread))