Ben Griffiths
04/29/2026, 8:59 AMMarvin
04/29/2026, 8:59 AMMarvin
04/29/2026, 9:02 AMThreadPoolTaskRunner.
That pattern can sometimes work, but it tends to produce exactly the kind of confusing behavior you're seeing:
- RolledBack states
- odd lineage / DAG rendering
- inner subflow tasks appearing attached to the wrong parent
Recommended rule of thumb:
- tasks call regular Python code
- flows call tasks and subflows
So instead of:
@task
def my_task(...):
return my_subflow(...)
prefer:
@flow
def parent():
...
result = my_subflow(...)
If you want concurrent subflows, the cleaner pattern is to orchestrate them from a flow, not from tasks.
For example, in Prefect 3.x, an async parent flow is usually the best option:
import asyncio
from prefect import flow, task
@task
async def inner_task(x):
return x * 2
@flow
async def child_flow(x):
return await inner_task(x)
@flow
async def parent_flow():
coros = [child_flow(x) for x in range(10)]
results = await asyncio.gather(*coros)
return results
Why this helps:
- subflows stay in flow orchestration context
- task/subflow relationships are tracked more cleanly
- you avoid mixing task-run transaction handling with nested flow creation inside worker threads
About the RolledBack states specifically: those usually come from Prefect’s transaction machinery when work inside that task/subflow stack fails or is unwound. When you nest a subflow inside a concurrently running task, the rollback/parent-child bookkeeping can get messy.
So my advice is:
1. Do not invoke subflows from tasks if you can avoid it
2. If you need concurrency for subflows, run them from a parent flow
3. If you need ThreadPoolTaskRunner, use it for tasks, not for tasks that themselves launch subflows
If you want, I can help rewrite your current pattern into one of these safer structures:
- concurrent tasks only
- concurrent subflows from a parent flow
- hybrid pattern with bothBen Griffiths
04/29/2026, 10:46 AMMarvin
04/29/2026, 10:46 AMMarvin
04/29/2026, 10:46 AM@task, you should treat it like normal Python code execution. That means:
- don’t call .submit() on other tasks from inside a task
- don’t orchestrate other tasks or subflows from inside a task
So this is not the recommended pattern:
from prefect import flow, task
@task
def inner(x):
return x * 2
@task
def outer(x):
return inner(x) # not recommended as a Prefect task call inside a task
@flow
def my_flow():
return outer(1)
Why:
- Prefect orchestration is designed around flows orchestrating tasks
- task run context / dependency tracking gets ambiguous inside tasks
- you can see weird DAG structure, state propagation, and rollback behavior
What is okay:
- calling a plain Python helper function from a task
def helper(x):
return x * 2
@task
def outer(x):
return helper(x)
- orchestrating tasks from a flow
from prefect import flow, task
@task
def inner(x):
return x * 2
@flow
def my_flow():
futs = [inner.submit(i) for i in range(5)]
return [f.result() for f in futs]
So the safe mental model is:
- flows orchestrate
- tasks execute work
- helper functions compose within tasks
If you want, paste a small version of your current flow/task structure and I can show the Prefect-native way to refactor it.