Can we schedule each task at a different time insi...
# prefect-community
m
Can we schedule each task at a different time inside a single flow? Example: a Flow is cronned for 3pm and it has TASK A, B, C. Task A should start at 3 pm but Task B and Task C should start at 4:30 pm and 5pm respectively.
s
I don't think you can do that currently. I suppose you could add a task that waits a given amount of time or until the required time before being successful.
a
It's the first time I see someone using "cron" as a verb 😄 I believe that introducing dependencies between tasks/flows in such a manner i.e. task A runs at 3 pm, task B runs at 3.30 pm is: • wasteful from a computation perspective - what if your task A doesn't need 30 min and instead finishes after 10 min? if all you need is ensuring that task B runs after task A, you can use Prefect to set dependencies between those in a much more robust way e.g. using the
upstream_tasks
keyword on the task • fragile - what if your task A had some network issues and need more than 30 min? setting dependencies in this time-based manner is quite error-prone - it introduces the risk that your task B starts too early because task A hasn't finished yet. But if you need these time-based dependencies for some reason, here are two options: 1) Manually setting time.sleep() One hack to delay the execution of specific tasks would be to add
time.sleep(3600)
e.g. to ensure that the next task runs one hour after the previous one finished. 2) Flow-of-flows Using the flow-of-flows orchestration pattern, you could trigger several child-flows from a parent flow and you can add a custom
scheduled_start_time
on the
create_flow_run
task:
Copy code
from datetime import timedelta
from prefect import Flow
from prefect.tasks.prefect import create_flow_run, wait_for_flow_run


with Flow("parent_flow") as flow:
    child_flow_run_id = create_flow_run(
        flow_name="child_flow_name",
        run_name="custom_run_name",
        scheduled_start_time=timedelta(minutes=30),
    )
    child_flowrunview = wait_for_flow_run(
        child_flow_run_id, raise_final_state=True, stream_logs=True,
    )
k
Anna is right that using
create_flow_run
to schedule sounds like a waste of compute. I think you may as well make those flows with a schedule.