https://prefect.io logo
#prefect-community
Title
# prefect-community
m

Matt Delacour

07/20/2022, 2:37 PM
Hi, I am running into issues with
LocalDaskExecutor
and I cannot understand the problem. I am trying to run parallel calls for Rest API endpoints. The problem is that the Dask executor will run some calls twice for no apparent reason. Here is the logic I have Then I can see that some Dask tasks run on the same endpoint while when I log all the endpoints, there are all unique... I will post the logic as a snippet in the thread 🧵
Untitled.py
Also if I run the same script with Dask, the bug is fixed. So pretty sure the problem comes from the DaskExecutor
k

Kevin Kho

07/20/2022, 4:31 PM
I think this can happen if a worker dies and tries to re-run the work it had. It won’t actually re-run though since it sees the completed state. Is this for LocalDask or Dask?
m

Matt Delacour

07/20/2022, 4:50 PM
It's LocalDask
So except if the worker here is a process, it might not apply
Also the LocalDask executor works well locally on my computer but has this weird bug when running on the DockerRun config in a remote server
k

Kevin Kho

07/20/2022, 6:17 PM
That’s pretty weird. I assume though you are not seeing extra tasks right in the mapped count? Just extra logs?
Ah I know the issue. I think your class is not thread safe
This is a good read
m

Matt Delacour

07/20/2022, 6:26 PM
Ah I know the issue. I think your class is not thread safe
Ha ... That explains it then. Maybe it would be worth adding a "Warning" section in the documentation about that specific issue. Lost couple of hours on that 😢
So I guess using a function should be thread safe though. Will have a look at that
k

Kevin Kho

07/20/2022, 6:44 PM
Or don’t use self inside your class. Take in the id instead
8 Views