<@ULVA73B9P> what happen if i got one task that us...
# ask-marvin
a
@Marvin what happen if i got one task that use .submit() and its called by other task that use .submit() its like doble concurrency, what i can except? a concurrent task that has inside a concurrent task
m
thought for 72.0 seconds
Here's what happens when you have nested
.submit()
calls: 1. Concurrent Execution: Yes, both tasks will run concurrently. When Task A uses
.submit()
to run Task B, and Task B itself uses
.submit()
to run Task C, you're effectively creating a hierarchy of concurrent tasks. 2. Thread Usage: By default, Prefect uses the
ThreadPoolTaskRunner
, so each submitted task runs in its own thread. With nested submits, you're creating multiple threads - one for each submitted task. Here's an example to illustrate:
Copy code
python
from prefect import flow, task

@task
def inner_task():
    return "inner result"

@task
def middle_task():
    # This task submits another task
    future = inner_task.submit()
    # Do other work concurrently while inner_task runs
    return future.result()

@flow
def main_flow():
    # This creates the first level of concurrency
    future = middle_task.submit()
    # Do other work concurrently while middle_task runs
    return future.result()
Important Considerations: 1. Resource Management: Be careful with deeply nested
.submit()
calls as each creates a new thread. Too many concurrent threads can impact performance or hit system limits. 2. Future Handling: Each
.submit()
returns a future object. You need to call
.result()
when you actually need the result, which will wait for the task to complete. 3. Dependencies: Despite running concurrently, tasks will still respect their dependencies. If Task B needs Task A's result, it will wait for Task A to complete regardless of
.submit()
. 4. Error Propagation: Errors in nested submitted tasks will propagate up through the futures chain when you call
.result()
. The key advantage of this pattern is that it allows for true concurrent execution of tasks, where both the parent and child tasks can perform work simultaneously. However, use this pattern judiciously and consider the resource implications of creating multiple concurrent threads.