Hi, I am running into issues with

LocalDaskExecutor

and I cannot understand the problem. I am trying to run parallel calls for Rest API endpoints. The problem is that the Dask executor will run some calls twice for no apparent reason. Here is the logic I have Then I can see that some Dask tasks run on the same endpoint while when I log all the endpoints, there are all unique... I will post the logic as a snippet in the thread 🧵

I think this can happen if a worker dies and tries to re-run the work it had. It won’t actually re-run though since it sees the completed state. Is this for LocalDask or Dask?

It's LocalDask

That’s pretty weird. I assume though you are not seeing extra tasks right in the mapped count? Just extra logs?

Ah I know the issue. I think your class is not thread safe

Ha ... That explains it then. Maybe it would be worth adding a "Warning" section in the documentation about that specific issue. Lost couple of hours on that 😢

Or don’t use self inside your class. Take in the id instead