09/16/2020, 10:18 PM
šŸ‘‹ hello from Chicago! At Saturn Cloud (, we offer a managed solution for data scientists and data engineers to easily provision and manage Dask clusters, so they can speed up their Python workflows. We've recently added an integration with Prefect Cloud, and I'm really excited to share it here! šŸŽ‰ The workflow we've designed is like this: ā€¢ Provision a
with a few clicks in the Saturn UI ā€¢ Author your flow in a Saturn-managed instance of Jupyter Lab ā€¢ Register the flow with Prefect Cloud. A library called
( adds an environment and storage to your flow that says "hey, run this on one of my Saturn Dask clusters" ā€¢ Whenever your flow runs, it creates a Dask cluster for itself programmatically. All your dependencies, files, and necessary credentials will be available on all of the Dask workers, and all logs and task statuses get sent back to Prefect Cloud. More details about the architecture and links to sample code are available at I want to thank the
team for the care and attention they've given to keeping
modular. We were able to write a client library like this (and another thin wrapper around
) without needing to manage a lot of code, because of the clear separation of concerns in the
library. The maintainers have also been very responsive to my questions in this Slack and to my proposals in issues and pull requests. ā¤ļø
šŸ™ 4
šŸŽ‰ 14
šŸ¤˜ 1
marvin 6
šŸ‘ 2