Running into a dilemma where I want to use Prefect for long-running, scraping orchestration. However, Prefect has a hard 24 hour limit for the duration job which is problematic. And, after breaking this into event-driven, smaller jobs, the Prefect event API has a limit that disables my automations and silently causes failures. Unsure the best solution from here. Any suggestions?

Hey @Bryant Richardson! What infrastructure are you using for your flows?

So we have Prefect Cloud but we run all flows on our own GCP infra

Gotcha, that 24 hr limit is on Cloud Run’s end. There’s no limit for how long flows can be. You could set up a process worker on a cheap VM is you want to run flows longer that 24 hours.

Okay thank you very much! I will look into that.

Connected question, as we do have a similar use case. We do have our short lived data pipelines running on prefect, but now we also want to move the scrapers, mostly just for observability and aligning on the tech stack. We are running a self hosted prefect with docker for deployments, is there any downside of having long running tasks for hours or days in this setup?

@Viktor, there's nothing stopping you from running days-long flows, but you will want to break them into tasks and persist results for those tasks so you can resume from failure.

Following up on this. So we set up these pipelines years ago. I think there was a point where cloud run only allowed 1 day max. But it looks like thats been upped to 7 days. I've updated my worker/work pool template to account for this. Unsure if its worth updating in the Prefect repo but here is the template I had to update.

Ah cool. The base job templates for work pools are designed to be flexible in that way, so that makes a lot of sense as a fix. If you want to submit a PR to update the default base job template, I'd be happy to review it!

😅 Large repo so I tried to track down the relevant parts: https://github.com/PrefectHQ/prefect/pull/21140