<@ULVA73B9P> Is there a way to create a scheduled ...
# ask-marvin
f
@Marvin Is there a way to create a scheduled deployment that are set to run at a specific time, but with a bit of jitter added? The usecase is that we are scheduling a large number of flows that end up overloading aws ecs task creation, spin up to many machines etc. I "think?" I could limit at workpool, or by creating multiple schedules and spread the load over a multiple schedules set within 10 minutes, but not at the exact same time. Ideally I want to have a schedule that starts at ~07:00 + 0-10 minutes or similar.
m
thought for 370.8 seconds
Short answer: there isn’t native “jitter” in Prefect schedules. In Prefect 3.x, cron syntax like “H”/“R” hashed randomization isn’t supported, and schedules don’t have a
jitter
option. See Schedules. Good news: there are a few solid patterns to prevent the ECS burst and spread starts within a window. Recommended options 1) Cap how many flows can start at once (best for ECS overload) - Work pool concurrency limit: This prevents your worker from starting too many flow runs (and thus ECS tasks) at once; excess runs queue up and are picked up gradually. - CLI (verified):
Copy code
prefect work-pool set-concurrency-limit "my-pool" 10
- Docs: Work pools - Optional: use queue-specific limits/priorities under that pool to trickle different cohorts while respecting the overall cap. - Docs: Work queues This directly addresses “too many ECS tasks created at the same time” regardless of schedule alignment. 2) Controller flow that creates daily runs with a randomized scheduled_time If you truly want “~07:00 + 0–10 minutes” for each run, create a small controller flow that, once per day, calls
run_deployment(..., scheduled_time=...)
with a randomized offset. Disable built-in schedules on the target deployment(s) so you don’t double-schedule. Example controller flow
Copy code
from datetime import datetime, timedelta
from zoneinfo import ZoneInfo
import random

from prefect import flow
from prefect.deployments import run_deployment

# Add one or many deployment names you want to start around 07:00
TARGET_DEPLOYMENTS = [
    "project-a/my-flow-a",
    "project-b/my-flow-b",
]

@flow
def schedule_with_jitter(
    base_time="07:00",
    tz="UTC",
    window_minutes=10,
    deterministic=False,
):
    # Compute today's base time in your timezone
    zone = ZoneInfo(tz)
    today = datetime.now(zone).date()
    hour, minute = map(int, base_time.split(":"))
    base_dt = datetime(today.year, today.month, today.day, hour, minute, tzinfo=zone)

    for name in TARGET_DEPLOYMENTS:
        rng = random.Random(
            hash((name, today.toordinal())) & 0xFFFFFFFF
        ) if deterministic else random

        offset = timedelta(seconds=rng.randint(0, window_minutes * 60))
        scheduled = base_dt + offset

        # Create a run that will start later (worker will launch the infra near this time)
        run_deployment(name=name, scheduled_time=scheduled)

# Serve this controller to run a bit before 07:00 every day
# e.g., a Cron("55 6 * * *", timezone="UTC") schedule for this flow
Notes: -
scheduled_time
is honored by Prefect’s scheduler; your worker will start the infrastructure (ECS task) near that time, so starts get spread across the window. - If you want stable-yet-varied offsets per deployment per day, set
deterministic=True
. 3) Staggered schedules/cohorts (simple, but fixed offsets) If randomness isn’t required, you can split runs across fixed offsets using multiple schedules (e.g., :00, :03, :06, :09). This is easy, but not random day-to-day. - Docs: Associate parameters with schedules Caveats - In-flow `sleep`/random delays do not reduce ECS bursts. The container (ECS task) is already started; the delay only pushes work inside the container. If the goal is to reduce task creation spikes, use work pool/queue limits or
scheduled_time
as above. - Deployment concurrency limits help bound concurrent runs of a single deployment, but won’t help when many different deployments all fire once at 07:00. If you share: - Cloud or OSS? - How many flows/deployments start at 07:00? - Your target max concurrent ECS tasks (or ramp-up rate)? …I can suggest concrete limits (work pool/queue) and a tailored controller flow setup. Helpful links
- Schedules: Docs - Work pools and concurrency: Docs - Global concurrency limits (for API/db rate limiting at task level): Docs