https://prefect.io logo
Title
j

Jeff Hale

03/07/2023, 12:54 PM
😛arty-parrot:Come learn how to use Dask to scale the Python tools you love! Join @Matthew Rocklin, creator of Dask and CEO of Coiled, this Wed., March 8, for an introduction. Streaming live at 3pm ET on Twitter, Twitch, and YouTube! 🎉
🙌 4
🔥 3
n

Nimesh Kumar

03/09/2023, 5:20 AM
Can someone please send the youtube link of this video
1
b

Brian Donaghy

03/09/2023, 5:28 AM

https://www.youtube.com/watch?v=zTkqlOR8YYY

n

Nimesh Kumar

03/09/2023, 5:29 AM
Thanks Brian.
👍 1
Hi everyone, Thanks for session, I have few question 1. Instead of parallelizing computation in a for loop, will it be possible run and scale the prefect flows using dask and coiled (for example i want to run 100 flows concurrently), if yes then is there any example that i can refer for how i can configure dask and coiled in my prefect flows. 2. How can i bring or use dask jupyterlab extension to my above mentioned scenario where i just have .py files.
1
j

Jeff Hale

03/09/2023, 2:20 PM
1. You can run 100 flows in parallel using Dask with a for loop. See the example in the prefect-dask collection - just click on After to see how to add the
DaskTaskRunner
and the
submit
method. The second example on that page shows how to connect to a running Dask Cluster - like one that could be provided by Coiled. See more about task runners in the Prefect docs. 2. You can copy Python code into a Jupyter Notebook and run it in Jupyter Lab.
n

Nimesh Kumar

03/09/2023, 3:15 PM
I think, i didnt phrase the question correctly 1. Let suppose 2 layer architecture "Layer A" and "Layer B". Layer B is where inferencing happens. The job of Layer A is to take the input and trigger the prefect flow using deployment I'd. So, Inference will be triggered as inputs comes in. So I don't see how I can put it in the fir loop. Yes if i already had 100 inputs for inferencing then i could have put them into a for loop. 2. Also in the link you have mention, i think there's tasks that are running in parallel because the flow name is same and i want to run flows in parallel
1
j

Jeff Hale

03/09/2023, 5:00 PM
Hi Nimesh. You don’t need to send to the channel. 1. An event comes in and you want to kick off a run. A second event comes in and you want to run it in parallel on the same Dask cluster. If so, you could probably just have a Dask cluster running and submit to it. Alternatively, in Prefect Cloud Automations, there are webhook enhancements coming soon to triggers. You could then kick off a deployment based on that trigger. 2. Flows running in parallel - you could have multiple agents running flows in parallel. They just need to pull from the same work pool. They could run outside of Dask. Alternatively, you could wrap a flow in a task and submit to the DaskTaskRunner.
n

Nimesh Kumar

03/10/2023, 4:13 AM
Thanks Jeff, By "If so, you could probably just have a Dask cluster running and submit to it" you meant
res_1 = generate_jobID.submit(algo_id, job_id)
res_2 = get_file.submit(file_path)
res_4 = choose_valid_file.submit(prev_task=res_2)

With task runner = DaskTaskRunner
if yes, I might be wrong here but what i read till now this will make task run in parallel (referring this https://docs.prefect.io/tutorials/dask-ray-task-runners/#running-parallel-tasks-with-dask) but not the flows.
j

Jeff Hale

03/10/2023, 1:41 PM
Right. Prefect doesn’t have a special mechanism to run flows in parallel - beyond wrapping them in tasks. But you should be able to just have multiple agents running, waiting for deployments, and submit the deployments to the same dask cluster.