Hi all
I'm trying to run a long running task that takes in a local DaskCluster but it seems like something on prefect arbitrarily kills off the worker... My flow uses the Dask ResourceManager looks something like
Copy code
@task(checkpoint=False)
def long_running_dask_task(inputs: dict, client: Client) -> boolean:
futures: List[Future] = []
while(True):
next = get_next(inputs)
if not next:
break
f = client.submit(func, next)
futures.append(f)
client.gather(futures)
return True
with Flow('local-dask-flow') as flow:
with DaskCluster(...) as client:
long_running_dask_task(param_1)
flow.executor = LocalExecutor()
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.