Andres Ferro
07/02/2025, 4:40 PMMarvin
07/02/2025, 4:40 PMMarvin
07/02/2025, 4:41 PM.submit()
calls:
1. Concurrent Execution: Yes, both tasks will run concurrently. When Task A uses .submit()
to run Task B, and Task B itself uses .submit()
to run Task C, you're effectively creating a hierarchy of concurrent tasks.
2. Thread Usage: By default, Prefect uses the ThreadPoolTaskRunner
, so each submitted task runs in its own thread. With nested submits, you're creating multiple threads - one for each submitted task.
Here's an example to illustrate:
python
from prefect import flow, task
@task
def inner_task():
return "inner result"
@task
def middle_task():
# This task submits another task
future = inner_task.submit()
# Do other work concurrently while inner_task runs
return future.result()
@flow
def main_flow():
# This creates the first level of concurrency
future = middle_task.submit()
# Do other work concurrently while middle_task runs
return future.result()
Important Considerations:
1. Resource Management: Be careful with deeply nested .submit()
calls as each creates a new thread. Too many concurrent threads can impact performance or hit system limits.
2. Future Handling: Each .submit()
returns a future object. You need to call .result()
when you actually need the result, which will wait for the task to complete.
3. Dependencies: Despite running concurrently, tasks will still respect their dependencies. If Task B needs Task A's result, it will wait for Task A to complete regardless of .submit()
.
4. Error Propagation: Errors in nested submitted tasks will propagate up through the futures chain when you call .result()
.
The key advantage of this pattern is that it allows for true concurrent execution of tasks, where both the parent and child tasks can perform work simultaneously. However, use this pattern judiciously and consider the resource implications of creating multiple concurrent threads.