Thread
#prefect-community
    Matt Delacour

    Matt Delacour

    2 months ago
    👋 When I schedule a Flow every "12 hours" using the
    IntervalSchedule
    object. Would the flows start at different time or all at the same time ? What I am looking for is to distribute the load on our servers when flows run. And so a new flow should start at a random time and then follow the
    IntervalSchedule
    constraint.
    James Sopkin

    James Sopkin

    2 months ago
    Hi Matt, you could use an IntervalSchedule to achieve this. I believe the interval schedule would run every 12 hrs after registration. You could also try setting two cron schedules and specifying exact times if you wanted to distribute flows very evenly
    import pendulum
    from prefect.schedules import Schedule, clocks
    
    schedule = Schedule(clocks=clocks.CronClock(
                     "0 0 * * *",
                     start_date=pendulum.now("US/Central")
    )
    schedule2 = Schedule(clocks=clocks.CronClock(
                     "0 12 * * *",
                     start_date=pendulum.now("US/Central")
    )
    Matt Delacour

    Matt Delacour

    2 months ago
    Yes but Prefect makes things start at the hour. So when I look at my servers, they are busy from when the hour starts to 10min later. For example, it's busy from 1 pm to 1:10 pm and idle from 1:10 pm to 2 pm What I would like is Prefect to schedule my IntervalSchedule jobs totally randomly at first and then stick to it.
    The I get random OOM errors because 10 jobs started exactly at 1 pm
    This is for example a flow I have
    with Flow(
        "flow_name",
        schedule=IntervalSchedule(
            interval=timedelta(hours=12),
        ),
    ) as flow:
        ...
    Does that make sense to you ?
    Hi @James Sopkin Just want to make sure you saw my late reply 🙏
    James Sopkin

    James Sopkin

    2 months ago
    @Matt Delacour Apologies for the late reply. Prefect was out on a company retreat last week. Currently there is no feature for random times for interval schedules. You could try using filters to set specific times see here in the filters section -> https://docs.prefect.io/core/concepts/schedules.html#clocks
    So try spacing the flows out with different starting times to reduce the load on your infrastructure
    Matt Delacour

    Matt Delacour

    2 months ago
    So try spacing the flows out with different starting times to reduce the load on your infrastructure
    Yes I guess, I will go with a random function for the start time to schedule the jobs. Might want to do it deterministically (eg: using the name of the flow)