Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

<@U03U4BQ9G6A> thats definitely how i use prefect. i create and use dask clusters in most of my flows that require any sort of meaningful scale

Got it. So the Prefect flow would have a single task that would start the compute on a dask cluster and monitor it?

yep. i use prefect for what its great at. being a really thin orchestration wrapper for logging, retries, state management, etc.

in cases where you are connecting multiple etls/bigger systems you can have multiple tasks or flows with subflows

but for parallelization like you're talking about, prefect doesnt need to be involved beyond executing the script/task and monitoring logs and exit status

Makes sense got it. Thank you for the input!