Robert
09/09/2024, 4:26 PMMarvin
09/09/2024, 4:27 PMwait_for
parameter. This parameter allows you to specify upstream dependencies, ensuring certain tasks or entire flows wait for others to complete before proceeding.
Here are some examples and important points:
### Using wait_for
in Tasks and Flows
You can make tasks or subflows wait for other tasks' results by using the wait_for
parameter.
python
from prefect import flow, task
@task
def task_a():
print("Task A completed")
@task
def task_b():
print("Task B completed")
@flow
def dependent_flow():
task_a_result = task_a.submit()
task_b_result = task_b.submit(wait_for=[task_a_result])
In this snippet:
- task_b
will wait for task_a
to complete before running.
### Pausing Flow Execution
You can pause a flow and resume it later, a method useful for out-of-process suspensions.
python
from prefect import flow, pause_flow_run, resume_flow_run, task
@task(persist_result=True)
def example_task():
return 42
@flow(persist_result=True)
def pausing_flow():
result = example_task.submit()
pause_flow_run(timeout=30, reschedule=True)
# Wait for the flow to resume
example_task(wait_for=[result])
- Use pause_flow_run
to suspend a flow, which can later be resumed using resume_flow_run
.
### Subflows and Upstream Dependencies
Upstream dependencies apply to all tasks, subflows, and mapped tasks submitted.
python
from prefect import flow, task
@flow
def parent_flow():
result = upstream_flow.submit()
downstream_flow.submit(wait_for=[result])
@flow
def upstream_flow():
print("Upstream flow")
@flow
def downstream_flow():
print("Downstream flow, waits for upstream flow")
This way, downstream_flow
will wait for the upstream_flow
to complete.
For more details, check out the Prefect 2.x documentation on flows and the guide on specifying upstream dependencies.
Hope this helps!Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by