<@ULVA73B9P> Is there such a thing as a time-based...
# ask-marvin
j
@Marvin Is there such a thing as a time-based concurrency limit on tasks or flows? I need to do a large backfill - I want the concurrency on the tasks (or could be flows if that would work better) to be higher at night and on weekends and less during working hours when other work is happening....
m
thought for 124.2 seconds
Short answer: not natively by time window. In Prefect 3.x, there are concurrency limits for tasks and flows, but there is not a built-in “higher at night / lower during business hours” scheduler for those limits. What does exist: - Global concurrency limits — adjustable limit you can apply around arbitrary code/resources - Tag-based task concurrency limits — for tasks with a given tag - Work pool / work queue / deployment concurrency limits — for flow-run execution capacity So for your use case, the usual pattern is: 1. Put the backfill work behind a concurrency limit 2. Change that limit on a schedule using: - a small scheduled Prefect flow, or - an external cron / scheduler Best fit for your case - If you want to throttle specific backfill tasks, use a global concurrency limit or tag-based task limit - If you want to throttle whole flow runs, use work queue / deployment / work pool concurrency For “more at night, less during working hours,” I’d usually recommend: - separate backfill flows/tasks from production ones - apply a named global concurrency limit to the backfill path - run a tiny scheduled job that updates the limit, e.g.: - weekdays 9–5 => limit 2 - nights/weekends => limit 20 CLI support for global concurrency limits is available: -
prefect gcl create
-
prefect gcl update
-
prefect gcl enable
-
prefect gcl disable
I verified the CLI exists:
Copy code
prefect gcl --help
prefect gcl update --help
Example idea:
Copy code
bash
prefect gcl update backfill-limit --limit 2
Then later:
Copy code
bash
prefect gcl update backfill-limit --limit 20
If you want to do it in Python, you can update the limit via the Prefect client as well. Helpful docs: - Global concurrency limits - Task run concurrency limits - Work pools - Work queues A couple practical patterns: Option 1: Scheduled limit changer - Schedule a small flow every hour - It checks current local time/day - It calls Prefect API to set
backfill-limit
to the right value Option 2: Separate queues - Put backfills on their own work queue - Raise/lower queue concurrency over time, or run more/fewer workers against that queue at night Option 3: Two deployments - “daytime backfill” deployment with low concurrency - “offhours backfill” deployment with high concurrency - schedule them differently If you want, I can sketch the cleanest Prefect 3.x implementation for either: - task-level backfill throttling, or - flow/deployment-level backfill throttling
j
@Marvin If I pause my backfill work pool during certain hours, what happens to any flows that are running at that time? I'm assuming any flows that haven't started will just be left in the queue.
m
thought for 144.8 seconds
Yes — your assumption is right. If you pause a work pool in Prefect 3.x: - already-running flow runs keep running - flow runs that have not started yet will not be picked up - those not-yet-started runs will effectively stay in their scheduled/queued state until the pool is resumed So pausing a pool is basically a stop dispatching new work control, not a stop currently running work control. A simple mental model: - Running now? continues - Not claimed by a worker yet? waits - Pool resumed later? workers can start picking those runs up again This aligns with the work pool behavior in the docs: - Work pools One important nuance: - pausing the pool is pretty coarse-grained — it stops all new work delivery for that pool - so if production and backfill share the same pool, pausing it will block both - if you only want to suppress backfill during business hours, it’s usually better to put backfill in a separate work pool or work queue Related docs: - Work queues If you want, I can help you choose between: - pausing a dedicated backfill pool - using a separate backfill queue - dynamically changing concurrency limits instead