Hi all - a question.
Say I have a flow run that has a task runner, and in said flow run I have a sub-flow I execute. Is there a way to inherit the task runner from the parent flow?
I ask because my task runner is backed by a slurm job allocation managed via
dask_jobqueue
. It makes sense to me to avoid a whole new set of requests when resources have already been allocated for the blocked parent flow.
r
Ryan Peden
01/16/2023, 8:34 AM
Hi Tim! Something like this might do what you need:
Copy code
from prefect.context import get_run_context
# then, inside your parent flow
context = get_run_context()
task_runner = context.task_runner
result = my_subflow.with_options(task_runner=task_runner)(param1, param2)
t
Tim Galvin
01/16/2023, 8:39 AM
Awesome -- that looks exactly like what I need
Tim Galvin
01/16/2023, 8:41 AM
Out of curiosity, what is the life time of a task_runner, particularly one that has been spun up outside a flow? Are they killed automatically when a
flow
using it completes? Or do they just live until that are garbage collected by python?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.