David Ojeda
07/10/2019, 3:58 PMprefect.engine.executors.DaskExecutor
to add the task name as the key
parameter sent to the dask client:
import uuid
from typing import Any, Callable
from distributed import Future
from prefect.engine.executors import DaskExecutor
class MyDaskExecutor(DaskExecutor):
def submit(self, fn: Callable, *args: Any, **kwargs: Any) -> Future:
if 'key' not in kwargs and 'task' in kwargs:
kwargs['key'] = kwargs['task'].__class__.__name__ + '-' + str(uuid.uuid4())
return super().submit(fn, *args, **kwargs)
So far this is working as expected, with the exception of tasks that use the .map(...)
idiom, where the run_fn still appears with that name. I have looked further into prefect’s code but it seems that to achieve this I would need to create my own task runner. Since that seems a bit complicated, I just wanted to ask if there is another solution or if this is just a bad idea?