Fredrik Hoem Grelland
12/04/2025, 8:53 AMMarvin
12/04/2025, 8:54 AMMarvin
12/04/2025, 9:00 AMjitter option. See Schedules.
Good news: there are a few solid patterns to prevent the ECS burst and spread starts within a window.
Recommended options
1) Cap how many flows can start at once (best for ECS overload)
- Work pool concurrency limit: This prevents your worker from starting too many flow runs (and thus ECS tasks) at once; excess runs queue up and are picked up gradually.
- CLI (verified):
prefect work-pool set-concurrency-limit "my-pool" 10
- Docs: Work pools
- Optional: use queue-specific limits/priorities under that pool to trickle different cohorts while respecting the overall cap.
- Docs: Work queues
This directly addresses “too many ECS tasks created at the same time” regardless of schedule alignment.
2) Controller flow that creates daily runs with a randomized scheduled_time
If you truly want “~07:00 + 0–10 minutes” for each run, create a small controller flow that, once per day, calls run_deployment(..., scheduled_time=...) with a randomized offset. Disable built-in schedules on the target deployment(s) so you don’t double-schedule.
Example controller flow
from datetime import datetime, timedelta
from zoneinfo import ZoneInfo
import random
from prefect import flow
from prefect.deployments import run_deployment
# Add one or many deployment names you want to start around 07:00
TARGET_DEPLOYMENTS = [
"project-a/my-flow-a",
"project-b/my-flow-b",
]
@flow
def schedule_with_jitter(
base_time="07:00",
tz="UTC",
window_minutes=10,
deterministic=False,
):
# Compute today's base time in your timezone
zone = ZoneInfo(tz)
today = datetime.now(zone).date()
hour, minute = map(int, base_time.split(":"))
base_dt = datetime(today.year, today.month, today.day, hour, minute, tzinfo=zone)
for name in TARGET_DEPLOYMENTS:
rng = random.Random(
hash((name, today.toordinal())) & 0xFFFFFFFF
) if deterministic else random
offset = timedelta(seconds=rng.randint(0, window_minutes * 60))
scheduled = base_dt + offset
# Create a run that will start later (worker will launch the infra near this time)
run_deployment(name=name, scheduled_time=scheduled)
# Serve this controller to run a bit before 07:00 every day
# e.g., a Cron("55 6 * * *", timezone="UTC") schedule for this flow
Notes:
- scheduled_time is honored by Prefect’s scheduler; your worker will start the infrastructure (ECS task) near that time, so starts get spread across the window.
- If you want stable-yet-varied offsets per deployment per day, set deterministic=True.
3) Staggered schedules/cohorts (simple, but fixed offsets)
If randomness isn’t required, you can split runs across fixed offsets using multiple schedules (e.g., :00, :03, :06, :09). This is easy, but not random day-to-day.
- Docs: Associate parameters with schedules
Caveats
- In-flow `sleep`/random delays do not reduce ECS bursts. The container (ECS task) is already started; the delay only pushes work inside the container. If the goal is to reduce task creation spikes, use work pool/queue limits or scheduled_time as above.
- Deployment concurrency limits help bound concurrent runs of a single deployment, but won’t help when many different deployments all fire once at 07:00.
If you share:
- Cloud or OSS?
- How many flows/deployments start at 07:00?
- Your target max concurrent ECS tasks (or ramp-up rate)?
…I can suggest concrete limits (work pool/queue) and a tailored controller flow setup.
Helpful links