Channels
#prefect-community
Title
y
Yaron Levi
01/11/2023, 5:53 PMHi guys 👋
Someone have experience with the DaskTaskRunner() and runing tasks in parallel?
We can’t make tasks run in parallel.
I’ve detailed the problem here:
https://discourse.prefect.io/t/tasks-are-not-running-in-parallel-when-using-dasktaskrunner/2180/1
👀 1
a
Andrew Huang
01/11/2023, 5:56 PMDoes this work for you?
Copy code
import time
from prefect import flow, task
from prefect_dask import DaskTaskRunner
@task
def shout(number):
time.sleep(5)
print(f"#{number}")
@flow(task_runner=DaskTaskRunner)
def count_to(highest_number):
for number in range(highest_number):
shout.submit(number)
if __name__ == "__main__":
count_to(10)y
Yaron Levi
01/11/2023, 5:57 PMThis is exactly my setup, but let me try your code as is.
maybe I am missing something super small
a
Andrew Huang
01/11/2023, 5:57 PMmessage has been deleted
I tried to mimic your code
Copy code
import time
from prefect import flow, task
from prefect_dask import DaskTaskRunner
@task
def shout(number):
print(f"before sleep")
time.sleep(5)
print(f"#{number}")
@flow(task_runner=DaskTaskRunner(), log_prints=True)
def count_to(highest_number):
cats = ["fluffy", "loaf", "fat", "squishy"]
for cat in cats:
shout.submit(cat)
if __name__ == "__main__":
count_to(10)y
Yaron Levi
01/11/2023, 5:58 PMmm… and where do you run the agent?
a
Andrew Huang
01/11/2023, 5:59 PMI don’t have a deployment set up. are you able to run it locally first?
y
Yaron Levi
01/11/2023, 6:00 PMI haven’t tried to run it locally
let me try
ok, so locally it’s the same thing.
won’t parallel.
So when you run it locally it dose run in parallel right?
a
Andrew Huang
01/11/2023, 6:02 PM Copy code
import time
from prefect import flow, task
from prefect_dask import DaskTaskRunner
@task
def shout(number):
print(f"before sleep")
time.sleep(5)
print(f"#{number}")
@flow(task_runner=DaskTaskRunner(cluster_kwargs={"n_workers": 4, "threads_per_worker": 2}), log_prints=True)
def count_to(highest_number):
cats = ["fluffy", "loaf", "fat", "squishy"]
for cat in cats:
shout.submit(cat)
if __name__ == "__main__":
count_to(10)what if you explicitly set n workers and yes it runs in parallel for me
y
Yaron Levi
01/11/2023, 6:05 PMmm…
one second
👀 2
ok this code works in parallel localy 👍👍
will now try to run it on Render
✅ 1
got it (-:
it was that return_state=True!
after I removed it, we are back to parallel.
👀 1
a
Andrew Huang
01/11/2023, 9:08 PMInteresting… that seems like a surprise. I think we could potentially improve this. do you mind submitting an issue to prefect-dask?
y
Yaron Levi
01/11/2023, 9:38 PMdone!
a
Andrew Huang
01/11/2023, 9:38 PMthanks!
2 Views