Thread
#prefect-community
    Dotan Asselmann

    Dotan Asselmann

    10 months ago
    Hi I encountered interesting issue when running the same flow using local agent vs a k8s agent, it seems like the same code takes longer to run on local agent compared to the k8s job [all on the same cluster, machine type, and resource requirements]. is there anything on local agent configuration or run config that can explain the difference? it runs x4 slower
    Anna Geller

    Anna Geller

    10 months ago
    is there anything on local agent configuration or run config that can explain the difference?
    no, there is not 😄 let me test it with a sample flow and get back to you
    Manuel Gomes

    Manuel Gomes

    10 months ago
    Network locality and/or traffic conditions between either runner and some required resource?
    local contention for some resource...
    ... zombie orphans in the pipe - there, I'm out of speculation 🙂
    Dotan Asselmann

    Dotan Asselmann

    10 months ago
    we’re on k8s, i’m running a local-runner deployment vs a deployment of k8s agent. running a scheduled task that roughly does the same. this is the logs before and after the switch:
    i had 2 failing tasks during the switch but that’s on me 🙂
    Manuel Gomes

    Manuel Gomes

    10 months ago
    <levity>I blame the
    manipulative-chimpanzee
    , it is obviously a troublemaker</levity>.
    Dotan Asselmann

    Dotan Asselmann

    10 months ago
    😅
    Anna Geller

    Anna Geller

    10 months ago
    @Dotan Asselmann I benchmarked this just now. To make a fair comparison, I used the same storage for both agents - Github storage. Also, both agents are running on the same machine - my laptop. Flows:1. This flow running on a local Kubernetes cluster: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/github_kubernetes_run.py 2. This flow running on a local agent: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/github_local_run.py The LocalRun is consistently showing 3 sec runtime. The KubernetesRun is showing 3-4 sec.
    so at this time there seem to be basically almost no difference. @Manuel Gomes has made some good points that you can check, but I would blame the latency to:1. Regional distances 2. Other possible networking latency 3. Could it be you use a different storage for each? Some storage classes are faster to download than others, e.g. pulling an image from a remote registry usually takes longer
    Manuel Gomes

    Manuel Gomes

    10 months ago
    This would correspond to my uninformed expectations - that, ceteris paribus, local would beat clustered by the small overhead introduced by cluster operations overhead.
    Also another "a bit of a stretch" scenario.. when you did the local run, did you actually shut down the cluster you were not using? (assuming same hardware)
    Anna Geller

    Anna Geller

    10 months ago
    no, the cluster was still running. But both the local agent and kubernetes agent are lightweight processes that offload execution to yet another process (local Kubernetes job vs. local subprocess) so not sure how this affects things 🙂 but yeah, good point, we should do it actually to ensure “ceteris paribus” 👍
    Dotan Asselmann

    Dotan Asselmann

    10 months ago
    same storage [S3]. real mystery. i’ll continue to investigate
    Manuel Gomes

    Manuel Gomes

    10 months ago
    chuckles I'm sorry, Anna & Dotan- I know bugger-all about prefect (in fact struggling to learn) but I've been around distributed systems more than it might possibly be healthy, so... https://ably.com/blog/8-fallacies-of-distributed-computing is always close to mind.