We have an hourly job and we’ve used a schedule deployment with a cron expression. We are also using Render (Heroku alternative). In Render we have a background worker (that runs the Prefect agent) with a CRON expression similar to that of the deployment. This way the background worker wakes up just in time to process the upcoming schedules, and we don’t need to run the worker 24/7 and pay for it. But sometimes, the background worker may fail, or not wake up in time. For example, Render is having an outage. So in that case, when the background worker comes back online, it runs all the late schedules. But it runs them all, and immediately. Is there a way to somehow define a “late” run policy. So for example, when the worker is back online it won’t run the missed schedules, or maybe do run them, but in a “push” manner, which will push the upcoming schedules to make room for the lates ones, keeping the time spans equal between them.

Hi Yaron, I'm not sure about the time span between late flow runs, but seeing your work queue concurrency to 1 should prevent the agent from running the missed flows all at once. Another alternative since you are already using Cron scheduling on Render would be to tell your Render container to run the script containing your flow instead of starting an agent. You'd need to call your flow function in your script, something like:

from prefect import flow

@flow(name="My excellent flow")
def my_etl_flow()
  read_data_from_mongo()
  write_data_to_snowflake()

if __name__ == '__main__':
  my_etl_flow()

Then, when your Render container starts up, tell it to run something like

python my_script.py

instead of

prefect agent start

. If you set the

PREFECT_API_URL

and (if you're using Prefect Cloud)

PREFECT_API_KEY

environment variables, the flow will still show up in the Prefect UI the way it does now. Since this approach wouldn't use a deployment, you'd need to include your flow code into your container image (if I remember our conversation a couple of weeks ago correctly, I think you were going to use S3 storage.) It's difficult for me to say which approach is best for your situation, but running the flow directly instead of using the Prefect scheduler might be simpler in this case if you're only expecting the agent to start up, run this one flow, and then shut down again.

OK, this is very interesting 🤔 This could be amazing. But if I just run the flow code in the Render container like you’ve said, wouldn’t I’ll be missing all the robust features of Prefect. For example; • Retrying a flow • Retrying a task • Seeing upcoming schedules in the UI Isn’t the Prefect agent a big part of the architecture? If I understand the system, without an agent, no one will poll the Prefect Cloud for robustness/retries etc…

Hi Yaron, those are excellent questions! There are certainly trade-offs if you run the flow directly, so I'll address each of your points. When running a flow directly instead of using the Prefect scheduler: • Retrying flows still works • Retrying tasks still works • You will not see upcoming scheduled tasks in the UI. ◦ This also makes it more difficult to tell if your Render background worker is not working properly. But you can still look at your list of completed flow runs, and if you don't see new ones appearing as expected, you will know something has gone wrong with your Render worker. Another downside is that you won't be able to trigger flow runs manually via the Prefect UI. Running your flow directly means you give up some functionality, but might also make it simpler for your current use case. Eventually, when you add more flows that need to run on different schedules, it would likely make sense to run the Render worker 24/7 and let Prefect schedule all your flow runs. Since you already have a deployment and agent, the simplest solution is probably to change the flow run concurrency for your work queue to 1, which you can do through the UI. I just wanted to mention the option of running your flow directly in case it's better for your current needs.

mm… Regarding “Retrying flows still works”, I am not talking about a “nice” error which is some, let’s say, null reference inside the flow/task function. I am talking about an immediate fatal crush of the whole Render job (in the process level).