@Marvin Can a flow schedule a future run of itself?

In Prefect 3.x, a flow itself doesn't directly schedule future runs. Instead, scheduling is typically handled through deployment configurations, where you define a schedule using cron expressions or other scheduling options. This is done when you create or update a deployment, not from within the flow itself. The Prefect Scheduler service is responsible for creating these scheduled flow runs based on the defined schedule. It places them in a Scheduled state but doesn't execute them. You can configure schedules using the

prefect deploy

command or by editing the

prefect.yaml

file of a deployment. For more detailed scheduling options and configurations, you can refer to the Prefect Scheduling Documentation. If you need a flow to trigger another run programmatically, you might consider using Prefect's API to create a new flow run from within a flow, but this would require additional handling outside of standard scheduling.

@Marvin nah, you can there's 2 main ways. search docs • run_deployment in an on_completion hook • put a trigger that expects a

prefect.flow-run.Completed

on your flow

You're correct. There are indeed methods to schedule future runs of a flow from within itself in Prefect 3.x: 1. Using

run_deployment

in an

on_completion

hook: You can set up an

on_completion

hook in your flow to call

run_deployment

, which will trigger another run of the same or a different deployment. This is a programmatically defined way to create a new flow run upon the completion of the current one. 2. Using a

prefect.flow-run.Completed

trigger: You can set up a trigger that listens for the

prefect.flow-run.Completed

event. This trigger can be configured to initiate a new flow run when the current flow run completes. It's part of Prefect's event-driven architecture, allowing for reactive automation based on flow run states. For more details on setting up these configurations, you can explore the Prefect Automations and Triggers Documentation.

note that for that these options you'd have to do something to break the infinite loop

Interesting. I have logic that checks if a process is running, rather than exiting and just waiting for the next scheduled flow run, I'd like to schedule a flow run to run again after the average run-time of that process (+5min).

gotcha what's your infra? i took the question at face value but the best option id imagine depends on the infra

We're still testing things out, so it's all local process/docker, but we are targeting ECS Tasks. Since billing is for uptime, I don't want a task to stand up for an hour when it could probably just check for that process, schedule a run for when it's most likely to be done, and then pop back down.

are you using cloud? ECS push pools sound like a good option (using either of the 2 patterns i mentioned before)

Unfortunately not cloud, this is self-hosted. We're still making sure this will cover all our needs.

gotcha, then the traditional ECS work pool would work, but typically you want to keep that polling process alive so it can dispatch the flow run container to your cluster at any time https://docs.prefect.io/integrations/prefect-aws/ecs_guide

I'll have to look at the triggers. I can probably use some logic in there. It's not a huge deal if it just sees the process is running, bails, and waits until the next scheduled run. It would just be nice for it to be able to add a scheduled run. I could probably also use the API to add a scheduled run as well.

this is alluded to in that doc I shared, but you need a polling process somewhere (worker) asking the server if there are scheduled runs, whether you use

run_deployment

or the events system to create the scheduled run the scheduled run will just bounce around server side until there's a worker (on our side in push pools, on your side if traditional "hybrid" work pool) that sees the scheduled run and submits it to the runtime (your ECS cluster)

Gotcha. I appreciate it! I'll poke at it, and update this if I find a solution.

👍

Quick update: I ended up not doing this due to limitations in the data I received from the API I was working with. :(