hi all, question about the

ProcessPoolTaskExecutor

(or really, any of the parallel task runners). Since tasks can be nested inside of tasks, what's the execution model when i

task.submit()

a task, and it calls

another_task()

inside of it, where does

another_task

run? in the process that is handling

task

? i ask because it used to be that we needed to wrap tasks in flow runs to control which task runner to use

if you

__call__

a task it doesn't go another thread/process, it runs like a normal function (this is new in 3.x) when you

.submit

or map a task it uses the parent flow's task runner

that part makes sense, the bit that's confusing is a scenario like this:

@task
my_inner_task():
    return 1 + 1

@task
def my_outer_task():
    my_inner_task.submit()

@flow(task_runner=ProcessPoolTaskRunner()
def my_flow():
    my_outer_task.submit()

based on what you said,

my_inner_task

should use the parent flow's task runner, but

my_outer_task

is already running in a separate process? does that mean

my_inner_task

gets propagated back to the parent flow's task runner and effectively runs at the same "level" as

my_outer_task

(where level is the process pool created by the parent flow)

i'd expect inner task runs in the process created for outer task by my_flow’s task runner which happens to be process task runner

i think that's what happened. which is fine, and doesn't particularly bother me, its more just to understand if there's some kind of inheritance of task execution patterns. i agree that its probably better to use a subflow to denote exactly what i want inside. this is just so i can set my own expectations of how this works

makes sense! fwiw claude repros our expectations here while im out on mobile :) <reasoning> 1. my_outer_task.submit() → runs in Process A (from the pool) a. Inside Process A, my_inner_task.submit() → runs in Process B (DIFFERENT process from the pool) b. Both use the parent flow’s ProcessPoolTaskRunner (via duplicated instance in each process) The inner task effectively runs at the “same level” in terms of using the flow’s task runner, but in a DIFFERENT process than the outer task. Each subprocess gets its own duplicated task runner instance with the same configuration. </reasoning>

sounds good!