Definitely let me know if I can answer more questions, either here or via DM.

im also curious about what your DS workflow looks like locally and how smooth the transition is to running things in production. I'm trying to help our nascent DS program get off the ground (im a DE just helping out on the infra/engineering side) and want to hear what its like

thanks again for taking the time to answer my questions!

Also, I didn't answer this part of your quesiton:

"im also curious about what your DS workflow looks like locally and how smooth the transition is to running things in production."

Prefect is really flexible about how you run Flows, there are 3 modes that we run in: 1. Run the Flow directly in a notebook by passing the hostname:port of our Dask scheduler to Prefect's DaskExecutor (development) 2. Register the Flow with Cloud using S3 storage, passing the hostname:port of our Dask scheduler to Prefect's RemoteEnvironment then run Flows with Cloud (development) 3. Register the Flow with Cloud using Docker storage, passing the hostname:port of our Dask scheduler to Prefect's RemoteEnvironment then run Flows with Cloud (production) We set up our Dask cluster using Kubernetes (and the dask-kubernetes project) on AWS EKS. It also runs JupyterLab in the same k8s environment. That has worked very well for us. (e.g. we use cluster-autoscaler to scale nodes for Dask workers up and down dynamically.)

We use the fargate agent and have not needed the persistent storage yet.

the DS team is also seriously looking at kubeflow, i have less experience with it but it honestly seems like way too much for our scale

i love prefect but if i have to pay for things that airflow already provides, and considering we already have a robust airflow deployment, it seems maybe the best option is to just combine the two?

"if i have to pay for things that airflow already provides"

I totally get where you're coming from. While there's a lot of overlap on the surface, I think Airflow & Prefect are a bit apples & oranges. To be specific, we viewed Airflow as far less of a fit for data science where fast iterations during model development are critical, e.g. the Airflow scheduler typically has pretty high inter-task latency and doesn't easily support dynamic flow concepts like mapping. While there are things you can do to try to address that, being able to have very fast execution and very easy scalability (train/test in parallel) and then easily put the same Flow into production was a huge advantage. For us, I wanted to make sure we had a tool that did the most important job well (speed & scalability for data science) and then be able to address other aspects. (BTW, definitely not saying Prefect is for sure the right choice for your scenario, just trying to share our perspective and thoughts. It's interesting to talk to someone considering Prefect for data science since I think the majority of Prefect users are focused on DE/ETL. I actually think Prefect for DS is going to be huge and once people start to understand the advantages there will be a lot of adoption of it for DS.)

once we get EFS or Lustre on Fargate and they implement the planned decrease in startup they talked about at re:invent..gonna be 🚀

since everything is containerized anyway, i shouldn't have any sticky "it runs locally but not in prod" scenarios i'd think...