https://prefect.io logo
Title
y

Yaron Levi

01/11/2023, 5:53 PM
Hi guys 👋 Someone have experience with the DaskTaskRunner() and runing tasks in parallel? We can’t make tasks run in parallel. I’ve detailed the problem here: https://discourse.prefect.io/t/tasks-are-not-running-in-parallel-when-using-dasktaskrunner/2180/1
👀 1
a

Andrew Huang

01/11/2023, 5:56 PM
Does this work for you?
import time

from prefect import flow, task
from prefect_dask import DaskTaskRunner

@task
def shout(number):
    time.sleep(5)
    print(f"#{number}")

@flow(task_runner=DaskTaskRunner)
def count_to(highest_number):
    for number in range(highest_number):
        shout.submit(number)

if __name__ == "__main__":
    count_to(10)
y

Yaron Levi

01/11/2023, 5:57 PM
This is exactly my setup, but let me try your code as is.
maybe I am missing something super small
a

Andrew Huang

01/11/2023, 5:57 PM
message has been deleted
I tried to mimic your code
import time

from prefect import flow, task
from prefect_dask import DaskTaskRunner

@task
def shout(number):
    print(f"before sleep")
    time.sleep(5)
    print(f"#{number}")

@flow(task_runner=DaskTaskRunner(), log_prints=True)
def count_to(highest_number):
    cats = ["fluffy", "loaf", "fat", "squishy"]
    for cat in cats:
        shout.submit(cat)

if __name__ == "__main__":
    count_to(10)
y

Yaron Levi

01/11/2023, 5:58 PM
mm… and where do you run the agent?
a

Andrew Huang

01/11/2023, 5:59 PM
I don’t have a deployment set up. are you able to run it locally first?
y

Yaron Levi

01/11/2023, 6:00 PM
I haven’t tried to run it locally
let me try
ok, so locally it’s the same thing. won’t parallel.
So when you run it locally it dose run in parallel right?
a

Andrew Huang

01/11/2023, 6:02 PM
import time

from prefect import flow, task
from prefect_dask import DaskTaskRunner

@task
def shout(number):
    print(f"before sleep")
    time.sleep(5)
    print(f"#{number}")

@flow(task_runner=DaskTaskRunner(cluster_kwargs={"n_workers": 4, "threads_per_worker": 2}), log_prints=True)
def count_to(highest_number):
    cats = ["fluffy", "loaf", "fat", "squishy"]
    for cat in cats:
        shout.submit(cat)

if __name__ == "__main__":
    count_to(10)
what if you explicitly set n workers and yes it runs in parallel for me
y

Yaron Levi

01/11/2023, 6:05 PM
mm…
one second
👀 2
ok this code works in parallel localy 👍👍
will now try to run it on Render
1
got it (-: it was that return_state=True! after I removed it, we are back to parallel.
👀 1
a

Andrew Huang

01/11/2023, 9:08 PM
Interesting… that seems like a surprise. I think we could potentially improve this. do you mind submitting an issue to prefect-dask?
y

Yaron Levi

01/11/2023, 9:38 PM
done!
a

Andrew Huang

01/11/2023, 9:38 PM
thanks!