Hi I am running into issues with `LocalDaskExecutor` and I c

Question

Hi I am running into issues with `LocalDaskExecutor` and I cannot understand the problem I am trying to run parallel calls for Rest API endpoints The problem is that the Dask executor will run some calls twice for no apparent reason Here is the logic I have Then I can see that some Dask tasks run on the same endpoint while when I log all the endpoints there are all unique I will post the logic as a snippet in the thread thread

Matt Delacour · Answer

Untitled py

Matt Delacour · Answer

Also if I run the same script with Dask the bug is fixed So pretty sure the problem comes from the DaskExecutor

Kevin Kho · Answer

I think this can happen if a worker dies and tries to re run the work it had It won t actually re run though since it sees the completed state Is this for LocalDask or Dask

Matt Delacour · Answer

It s LocalDask

Matt Delacour · Answer

So except if the worker here is a process it might not apply

Matt Delacour · Answer

Also the LocalDask executor works well locally on my computer but has this weird bug when running on the DockerRun config in a remote server

Kevin Kho · Answer

That s pretty weird I assume though you are not seeing extra tasks right in the mapped count Just extra logs

Kevin Kho · Answer

Ah I know the issue I think your class is not thread safe

Kevin Kho · Answer

<https github com PrefectHQ prefect issues 4958 issue 993696540|This> is a good read

Matt Delacour · Answer

gt Ah I know the issue I think your class is not thread safe Ha That explains it then Maybe it would be worth adding a Warning section in the documentation about that specific issue Lost couple of hours on that cry

Matt Delacour · Answer

So I guess using a function should be thread safe though Will have a look at that

Kevin Kho · Answer

Or don t use self inside your class Take in the id instead