https://prefect.io logo
Title
j

james.lamb

09/16/2020, 10:18 PM
šŸ‘‹ hello from Chicago! At Saturn Cloud (https://www.saturncloud.io/s/), we offer a managed solution for data scientists and data engineers to easily provision and manage Dask clusters, so they can speed up their Python workflows. We've recently added an integration with Prefect Cloud, and I'm really excited to share it here! šŸŽ‰ The workflow we've designed is like this: • Provision a
KubernetesAgent
with a few clicks in the Saturn UI • Author your flow in a Saturn-managed instance of Jupyter Lab • Register the flow with Prefect Cloud. A library called
prefect-saturn
(https://github.com/saturncloud/prefect-saturn) adds an environment and storage to your flow that says "hey, run this on one of my Saturn Dask clusters" • Whenever your flow runs, it creates a Dask cluster for itself programmatically. All your dependencies, files, and necessary credentials will be available on all of the Dask workers, and all logs and task statuses get sent back to Prefect Cloud. More details about the architecture and links to sample code are available at https://www.saturncloud.io/docs/connecting/tools/prefect-cloud/. I want to thank the
prefect
team for the care and attention they've given to keeping
prefect
modular. We were able to write a client library like this (and another thin wrapper around
KubernetesAgent
) without needing to manage a lot of code, because of the clear separation of concerns in the
prefect
library. The maintainers have also been very responsive to my questions in this Slack and to my proposals in issues and pull requests. ā¤ļø
šŸ™ 4
šŸŽ‰ 14
🤘 1
:marvin: 6
šŸ‘ 2