Thread
#prefect-community
    Samuel Hinton

    Samuel Hinton

    1 year ago
    Hi all! Is there a setting I can pass to the agents to say “Poll every 2 seconds” or similar instead of every 10 seconds. 10 seconds is pretty good but we have a lot of quick tasks flows happening and picking them up faster would be great!
    Dylan

    Dylan

    1 year ago
    Hi @Samuel Hinton! I first want to clarify one thing: Agents aren’t responsible for the intra-Flow-Run scheduling. The Flow Run itself schedules all of its Task Runs. If you’re looking to get Tasks to execute more quickly, look into some of our executors that scale horizontally well and give you parallelism out of the box
    Samuel Hinton

    Samuel Hinton

    1 year ago
    Hi Dylan, yeah Im using a Dask executor right now and the intra-flow run is looking good. Its the polling for scheduled flows Id like to increase. I realised I said “tasks” when I meant polls in the original comment, my apologies!
    Dylan

    Dylan

    1 year ago
    No worries! “Task” is a broadly used term haha
    So you’re in luck! That 10-second backoff is only when no Flow Runs are being detected by the agent. Once a run is picked up, the Agent automatically decreases its polling interval (down to 0.25 seconds depending on how many runs are being returned with each call)
    So while it’s not directly configurable, the Agent should already be behaving as you expect 👍
    Samuel Hinton

    Samuel Hinton

    1 year ago
    Thats good to know, thanks. Are there any plans in having that 10second amount configurable via the command line? (Naively) it seems like an easy option to add, and would be nice to see so that my 2second duration flows spend more time executing than waiting to be executed 🙂
    Dylan

    Dylan

    1 year ago
    I don’t believe there are any plans at the moment. However, some of the time you may be seeing in the UI is could be the time to provision infrastructure. If, for example, you’re running with a Docker Run Config + a Docker Agent, we don’t count the time to pull and run the image into the execution time. Execution starts when your Flow’s script actually begins to run
    Samuel Hinton

    Samuel Hinton

    1 year ago
    Yup, I moved from the dockeragent to a local agent exactly for this - to have smaller and more rapid execution
    Dylan

    Dylan

    1 year ago
    If you believe you’d get a significant benefit from configuring the polling interval, I’d encourage you to open an issue or start a GitHub discussion with a description of:1. Your business case (i.e. what do your Flows do & any background on what you’re trying to achieve from a business sense) 2. Your technical usecase (what schedules/creates your Flow Runs, what your Flows do, etc) 3. Why you believe this configuration would be helpful to the above cases
    We’ll definitely consider any enhancement that will provide you with a benefit 🚀
    Samuel Hinton

    Samuel Hinton

    1 year ago
    Sure thing, I can see it would be a fairly simple change here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/agent.py#L233 So I can try and contribute some code as well
    Thanks for the help Dylan, Ive raised the issue here: https://github.com/PrefectHQ/prefect/issues/4253 with a tested code function to generate the _loop_intervals 🙂
    @Dylan / @Michael Adkins - just wanted to follow this up and ask if theres anything more I can add or do in the issue to increase the chance this is incorporated into the product. Unsure if its been seen at all on the github side, and Im happy to try and open a formal PR if that would help?