run getFromSimInMaster in parallel

[zepumph]

We could open more than one page in puppeteer.

[samreid]

It crossed my mind that parallel puppeteer may be significantly better than sequential aqua--maybe at some point we should redo aqua to use puppeteer?

[zepumph]

With how easy it was to set up puppeteer, I would definitely be interested in pursuing that.

[zepumph]

I would first want to see how nice it was for the binder work, as I think that will be a bit simpler.

[mbarlow12]

One big issue I'm running into is resource management. Generating a page per sim asynchronously works well when the number of sims is short; however, when trying to run through all sims, the same code performs terribly.

My first thought is to chunk the sims into manageable sizes. So puppeteer would only have ~10 pages active at a time and process each asynchronously then move on to the next.

Another though would be to gather the dataURLs from the demo sims instead of each individual sim. Pulling from the rendered sims results in a ton of images that are almost identical, while the demo sims provide a set of contrasting examples of each component. I assume we're also free to add more examples in joist/kite/scenery demos if necessary.

Getting the sim usage might be a little trickier, but traversing the file system or perhaps using the requirejs node package would likely be pretty quick. We wouldn't get sim-specific images, but we could get the links to sims at least.

[zepumph]

while the demo sims provide a set of contrasting examples of each component.

I thought the demo sim just has one example of it, and some components have dynamic controls to adjust some options.

Pulling from the rendered sims results in a ton of images that are almost identical,

I agree they are all mostly identical. Maybe there is a way to minimize this part of the documentation since it is by far the largest section of content for each component, but it isn't proportionally that important. I think it will still be good to see how different sims use the same component, if most are the same, that is a nice data point for a designer.

Getting the sim usage might be a little trickier, but traversing the file system or perhaps using the requirejs node package would likely be pretty quick.

@samreid and I thought about this originally, and it seemed like a whole can of worms. There are multiple ways that we use components in sims, so it is hard to search for them.

[mbarlow12]

I thought the demo sim just has one example of it

When I was looking through the sun demo, the number spinner presents several different appearance options. I realize that most of the components only present a single version; is there a reason we couldn't add other examples?

Another thought on a file system approach for which sims use which components is that it could run in parallel to the instance image fetching.

[zepumph]

If we are just worried about time, we could work to make sim loads on pupeteer. We are currently using a for ( let sim of sims ) { loop to load each sim one after another, which is sync for each sim. So slow!!! Instead why not explore having multiple (a) browser instances (like const browser = await puppeteer.launch();) (b) tabs?, (c) pages at the same time to load sims in parallel.

Sorry if we have already discussed this somewhere else and I just missed it.