🦜Come learn how to use Dask to scale the Python tools you love! Join @Matthew Rocklin, creator of Dask and CEO of Coiled, this Wed., March 8, for an introduction. Streaming live at 3pm ET on Twitter, Twitch, and YouTube! 🎉

Can someone please send the youtube link of this video

Thanks Brian.

1. You can run 100 flows in parallel using Dask with a for loop. See the example in the prefect-dask collection - just click on After to see how to add the

DaskTaskRunner

and the

submit

method. The second example on that page shows how to connect to a running Dask Cluster - like one that could be provided by Coiled. See more about task runners in the Prefect docs. 2. You can copy Python code into a Jupyter Notebook and run it in Jupyter Lab.

I think, i didnt phrase the question correctly 1. Let suppose 2 layer architecture "Layer A" and "Layer B". Layer B is where inferencing happens. The job of Layer A is to take the input and trigger the prefect flow using deployment I'd. So, Inference will be triggered as inputs comes in. So I don't see how I can put it in the fir loop. Yes if i already had 100 inputs for inferencing then i could have put them into a for loop. 2. Also in the link you have mention, i think there's tasks that are running in parallel because the flow name is same and i want to run flows in parallel

Hi Nimesh. You don’t need to send to the channel. 1. An event comes in and you want to kick off a run. A second event comes in and you want to run it in parallel on the same Dask cluster. If so, you could probably just have a Dask cluster running and submit to it. Alternatively, in Prefect Cloud Automations, there are webhook enhancements coming soon to triggers. You could then kick off a deployment based on that trigger. 2. Flows running in parallel - you could have multiple agents running flows in parallel. They just need to pull from the same work pool. They could run outside of Dask. Alternatively, you could wrap a flow in a task and submit to the DaskTaskRunner.

Thanks Jeff, By "If so, you could probably just have a Dask cluster running and submit to it" you meant

res_1 = generate_jobID.submit(algo_id, job_id)
res_2 = get_file.submit(file_path)
res_4 = choose_valid_file.submit(prev_task=res_2)

With task runner = DaskTaskRunner

if yes, I might be wrong here but what i read till now this will make task run in parallel (referring this https://docs.prefect.io/tutorials/dask-ray-task-runners/#running-parallel-tasks-with-dask) but not the flows.

Right. Prefect doesn’t have a special mechanism to run flows in parallel - beyond wrapping them in tasks. But you should be able to just have multiple agents running, waiting for deployments, and submit the deployments to the same dask cluster.