I’m evaluating Prefect along with Metaflow and Kubeflow Pipelines for 1) general ETL / analytics workflows and 2) machine learning training + deployment workflows. So far, I’m most excited about Prefect. Prefect’s native support for multiple environments (local, dask-local, dask-distributed, dask-kubernetes…) makes it accessible to anyone doing analytics at any scale in my company. It’s checkpointing feature makes debugging easier. The thing I like about Metaflow is that you can specify dependencies and resources at the task-level (ie run this task on AWS Batch with 2 GPUs), and seamlessly inspect run artifacts in a jupyter notebook. The thing I like about Kubeflow Pipelines (which we currently run in production) is how tasks can be authored, shared, and composed into new pipelines, and its UI supports arbitrary, rich output from each task. Does Prefect Core / Cloud envision supporting any of these features?

Hey Jake! Glad to hear Prefect resonated with you. With response to your two specific questions - we DO have a roadmap item for specifying per-task environments and while we have not considered arbitrary output, we have been talking about some form of “progress API” that would allow tasks to yield information purely for UI consumption. We’d love to hear more about your use case if you’re game.

Yeah totally — also I applied for Cloud and I’m excited to hear back about that.

Just to add on here: in addition to the progress API, we’ve also been playing around with an “artifacts” API for tasks which would allow you to publish various JSON information from your tasks which would then be inspectable in the UI

Hi Jake, pleasure to meet you! I just sent a few options over via email for us to connect. Looking forward to chatting!