<@ULVA73B9P> Can you call a sub-flow from a task w...
# ask-marvin
b
@Marvin Can you call a sub-flow from a task which is running concurrently with ThreadPoolTaskRunner? I get a lot of RolledBack states, meaning the inner tasks from the subflow are rendered as parent items in the DAG.
m
thought for 153.8 seconds
Short answer: you generally should not call a subflow from inside a task, especially not a task running concurrently in a
ThreadPoolTaskRunner
. That pattern can sometimes work, but it tends to produce exactly the kind of confusing behavior you're seeing: -
RolledBack
states - odd lineage / DAG rendering - inner subflow tasks appearing attached to the wrong parent Recommended rule of thumb: - tasks call regular Python code - flows call tasks and subflows So instead of:
Copy code
@task
def my_task(...):
    return my_subflow(...)
prefer:
Copy code
@flow
def parent():
    ...
    result = my_subflow(...)
If you want concurrent subflows, the cleaner pattern is to orchestrate them from a flow, not from tasks. For example, in Prefect 3.x, an async parent flow is usually the best option:
Copy code
import asyncio
from prefect import flow, task

@task
async def inner_task(x):
    return x * 2

@flow
async def child_flow(x):
    return await inner_task(x)

@flow
async def parent_flow():
    coros = [child_flow(x) for x in range(10)]
    results = await asyncio.gather(*coros)
    return results
Why this helps: - subflows stay in flow orchestration context - task/subflow relationships are tracked more cleanly - you avoid mixing task-run transaction handling with nested flow creation inside worker threads About the
RolledBack
states specifically: those usually come from Prefect’s transaction machinery when work inside that task/subflow stack fails or is unwound. When you nest a subflow inside a concurrently running task, the rollback/parent-child bookkeeping can get messy. So my advice is: 1. Do not invoke subflows from tasks if you can avoid it 2. If you need concurrency for subflows, run them from a parent flow 3. If you need
ThreadPoolTaskRunner
, use it for tasks, not for tasks that themselves launch subflows If you want, I can help rewrite your current pattern into one of these safer structures: - concurrent tasks only - concurrent subflows from a parent flow - hybrid pattern with both
b
@Marvin Calling tasks from within tasks is okay right?
m
thought for 12.3 seconds
Usually: no, that’s not a supported Prefect pattern either. Inside a
@task
, you should treat it like normal Python code execution. That means: - don’t call
.submit()
on other tasks from inside a task
- don’t orchestrate other tasks or subflows from inside a task So this is not the recommended pattern:
Copy code
from prefect import flow, task

@task
def inner(x):
    return x * 2

@task
def outer(x):
    return inner(x)  # not recommended as a Prefect task call inside a task

@flow
def my_flow():
    return outer(1)
Why: - Prefect orchestration is designed around flows orchestrating tasks - task run context / dependency tracking gets ambiguous inside tasks - you can see weird DAG structure, state propagation, and rollback behavior What is okay: - calling a plain Python helper function from a task
Copy code
def helper(x):
    return x * 2

@task
def outer(x):
    return helper(x)
- orchestrating tasks from a flow
Copy code
from prefect import flow, task

@task
def inner(x):
    return x * 2

@flow
def my_flow():
    futs = [inner.submit(i) for i in range(5)]
    return [f.result() for f in futs]
So the safe mental model is: - flows orchestrate - tasks execute work - helper functions compose within tasks If you want, paste a small version of your current flow/task structure and I can show the Prefect-native way to refactor it.