https://prefect.io
Join Slack
Hi guys :wave: Someone have experience with the DaskTaskRunner() and runing tasks in parallel? We ca...
# ask-community
y
Hi guys 👋 Someone have experience with the DaskTaskRunner() and runing tasks in parallel? We can’t make tasks run in parallel. I’ve detailed the problem here: https://discourse.prefect.io/t/tasks-are-not-running-in-parallel-when-using-dasktaskrunner/2180/1
👀 1
a
Does this work for you?
Copy code
import time

from prefect import flow, task
from prefect_dask import DaskTaskRunner

@task
def shout(number):
    time.sleep(5)
    print(f"#{number}")

@flow(task_runner=DaskTaskRunner)
def count_to(highest_number):
    for number in range(highest_number):
        shout.submit(number)

if __name__ == "__main__":
    count_to(10)
y
This is exactly my setup, but let me try your code as is.
maybe I am missing something super small
a
message has been deleted
I tried to mimic your code
Copy code
import time

from prefect import flow, task
from prefect_dask import DaskTaskRunner

@task
def shout(number):
    print(f"before sleep")
    time.sleep(5)
    print(f"#{number}")

@flow(task_runner=DaskTaskRunner(), log_prints=True)
def count_to(highest_number):
    cats = ["fluffy", "loaf", "fat", "squishy"]
    for cat in cats:
        shout.submit(cat)

if __name__ == "__main__":
    count_to(10)
y
mm… and where do you run the agent?
a
I don’t have a deployment set up. are you able to run it locally first?
y
I haven’t tried to run it locally
let me try
ok, so locally it’s the same thing. won’t parallel.
So when you run it locally it dose run in parallel right?
a
Copy code
import time

from prefect import flow, task
from prefect_dask import DaskTaskRunner

@task
def shout(number):
    print(f"before sleep")
    time.sleep(5)
    print(f"#{number}")

@flow(task_runner=DaskTaskRunner(cluster_kwargs={"n_workers": 4, "threads_per_worker": 2}), log_prints=True)
def count_to(highest_number):
    cats = ["fluffy", "loaf", "fat", "squishy"]
    for cat in cats:
        shout.submit(cat)

if __name__ == "__main__":
    count_to(10)
what if you explicitly set n workers and yes it runs in parallel for me
y
mm…
one second
👀 2
ok this code works in parallel localy 👍👍
will now try to run it on Render
✅ 1
got it (-: it was that return_state=True! after I removed it, we are back to parallel.
👀 1
a
Interesting… that seems like a surprise. I think we could potentially improve this. do you mind submitting an issue to prefect-dask?
y
done!
a
thanks!
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by