Commits on Jun 15, 2023

1. Experimenting with Dask integration …

   Adds two local plugins and associated infrastructure for most hooks to be able to use Dask.
   
   In most cases for different types of Dask infrastruture, a plugin that provides a `get_dask_cluster()` method should do the trick. The hook is set up to only return one result, and the built in plugin will be the last.
   
   The Dask client plugin in theory should work with different types of clusters, but is similarly set up to be able to be overridden (dask-on-ray?). The client can be both sync and async, and once it gets accessed, it's cached on `xpublish.Rest`.
   
   For hooks that have access to `deps` (which now includes dataset providers), `deps.dask_sync_client` and `deps.dask_async_client` now should give you the client.
   
   The async client may need to be passed the current event loop. It appears the way to access the event loop varies by server, so that will probably take some research.
   - fastapi/fastapi#7876
   - encode/uvicorn#706
   - https://stackoverflow.com/questions/66275747/how-to-use-event-loop-created-by-uvicorn

   

   abkfenris committed Jun 15, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for b81b392
   • Browse repository at this point

   Copy the full SHA

   b81b392 View commit details

   Browse the repository at this point in the history