Hi Prefect, I'm running into the following warning when running my flow:

/venv/lib/python3.9/site-packages/prefect/engine/task_runner.py:865: UserWarning: This task is running in a daemonic subprocess; consequently Prefect can only enforce a soft timeout limit, i.e., if your Task reaches its timeout limit it will enter a TimedOut state but continue running in the background.

I'm running a scheduled flow via a Docker Agent with DaskExecutors (multiple processes due to CPU intensive nature of the task). Some of the tasks in the flow can hang and thus I've added a timeout on the task. This warning scares me because, if I interpret it correctly, it might mean that certain processes might hang indefinitely. What is the best way to avoid this and still benefit from running multiple CPU intensive tasks in parallel with Prefect? (I'm running Prefect 0.14.19 and Dask 2021.5.0).

I see what you mean @Peter Roelants. If it hangs, what behavior do you want? Do you have other tasks downstream that you still want to run? Or do you want the whole flow to cancel?

Ideally, I want to make sure the process is killed. If that means cancelling the whole flow that's ok (I can rerun it afterwards).

Will ask the team about this

Thanks 🙏

So the quick answer is that Prefect does not handle shutting down work that has been sent to the Dask workers. This is because it’s very hard in general to shut things down on different machines is hard in Python.

What if this is a temporary Dask cluster spun up by prefect's DaskExecutor? (This is where this error comes from) In this case I don't have any control over the Dask workers myself. I guess killing the flow should kill the whole instance in this case?

Oh ok. Glad you figured it out. Also, I have seen switching from threads to processes help some users with Dask hanging.

Timeouts are a little tricky because we have to enforce them in different ways for different execution models; it may be a little low level but you can read the code at https://github.com/PrefectHQ/prefect/blob/master/src/prefect/utilities/executors.py#L283 and I've tried to make the comments explain what kind of use-case encounters each timeout handler

Thank you for the reference