Robert
09/09/2024, 4:26 PMMarvin
09/09/2024, 4:27 PMwait_for
parameter. This parameter allows you to specify upstream dependencies, ensuring certain tasks or entire flows wait for others to complete before proceeding.
Here are some examples and important points:
### Using wait_for
in Tasks and Flows
You can make tasks or subflows wait for other tasks' results by using the wait_for
parameter.
python
from prefect import flow, task
@task
def task_a():
print("Task A completed")
@task
def task_b():
print("Task B completed")
@flow
def dependent_flow():
task_a_result = task_a.submit()
task_b_result = task_b.submit(wait_for=[task_a_result])
In this snippet:
- task_b
will wait for task_a
to complete before running.
### Pausing Flow Execution
You can pause a flow and resume it later, a method useful for out-of-process suspensions.
python
from prefect import flow, pause_flow_run, resume_flow_run, task
@task(persist_result=True)
def example_task():
return 42
@flow(persist_result=True)
def pausing_flow():
result = example_task.submit()
pause_flow_run(timeout=30, reschedule=True)
# Wait for the flow to resume
example_task(wait_for=[result])
- Use pause_flow_run
to suspend a flow, which can later be resumed using resume_flow_run
.
### Subflows and Upstream Dependencies
Upstream dependencies apply to all tasks, subflows, and mapped tasks submitted.
python
from prefect import flow, task
@flow
def parent_flow():
result = upstream_flow.submit()
downstream_flow.submit(wait_for=[result])
@flow
def upstream_flow():
print("Upstream flow")
@flow
def downstream_flow():
print("Downstream flow, waits for upstream flow")
This way, downstream_flow
will wait for the upstream_flow
to complete.
For more details, check out the Prefect 2.x documentation on flows and the guide on specifying upstream dependencies.
Hope this helps!