Thread
#prefect-community
    Prabin Mehta

    Prabin Mehta

    1 year ago
    Hi Everyone. I am running prefect in a Kubernetes cluster. I am using Dask as executor and for the run environment using Kubernetes Run. I am using the cloud and in the trial. What I found is, I have a lag between 4-65 sec between job scheduled and job running. Is there any way we can tune the scheduler or the worker for running with minimum lags? Let me know if I have to check some config for this.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Prabin Mehta, unfortunately the answer here with Kubernetes is that there is an expected latency triggered by many factors, but not specifically related to Prefect, and usually beyond our control.
    Prabin Mehta

    Prabin Mehta

    1 year ago
    And is the lag same when using DaskExecutor?
    Kevin Kho

    Kevin Kho

    1 year ago
    DaskExectutor will have a lag if it needs to spin up and install images on workers. If you’re connecting to an existing cluster, there might be less.
    Zach Schumacher

    Zach Schumacher

    1 year ago
    as kevin said, going to be releated to if youre using spot instances, what your image pull policy is, how large your image is, how long your container takes to start up, etc.
    you’d want to look up how you can decrease k8s jobs start up as opposed to anything on the prefect side, as the prefect agent is just building a job template and sending it to your cluster via the k8s python sdk