https://prefect.io
Join Slack
Hi, I am running into issues with `LocalDaskExecutor` and I cannot understand the problem. I am tryi...
# ask-community
m
Hi, I am running into issues with 
LocalDaskExecutor
and I cannot understand the problem. I am trying to run parallel calls for Rest API endpoints. The problem is that the Dask executor will run some calls twice for no apparent reason. Here is the logic I have Then I can see that some Dask tasks run on the same endpoint while when I log all the endpoints, there are all unique... I will post the logic as a snippet in the thread 🧵
Untitled.py
Untitled.py
Also if I run the same script with Dask, the bug is fixed. So pretty sure the problem comes from the DaskExecutor
k
I think this can happen if a worker dies and tries to re-run the work it had. It won’t actually re-run though since it sees the completed state. Is this for LocalDask or Dask?
m
It's LocalDask
So except if the worker here is a process, it might not apply
Also the LocalDask executor works well locally on my computer but has this weird bug when running on the DockerRun config in a remote server
k
That’s pretty weird. I assume though you are not seeing extra tasks right in the mapped count? Just extra logs?
Ah I know the issue. I think your class is not thread safe
This is a good read
m
Ah I know the issue. I think your class is not thread safe
Ha ... That explains it then. Maybe it would be worth adding a "Warning" section in the documentation about that specific issue. Lost couple of hours on that 😢
So I guess using a function should be thread safe though. Will have a look at that
k
Or don’t use self inside your class. Take in the id instead
11 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by