Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi I encountered interesting issue when running the same flow using local agent vs a k8s agent, it seems like the same code takes longer to run on local agent compared to the k8s job [all on the same cluster, machine type, and resource requirements]. is there anything on local agent configuration or run config that can explain the difference?  it runs x4 slower

&gt; is there anything on local agent configuration or run config that can explain the difference?
no, there is not :smile: let me test it with a sample flow and get back to you

Network locality and/or traffic conditions between either runner and some required resource?

... zombie orphans in the pipe - there, I'm out of speculation :slightly_smiling_face:

image.png

we’re on k8s, i’m running a local-runner deployment vs a deployment of k8s agent.   running a scheduled task that roughly does the same.  this is the logs before and after the switch:

i had 2 failing tasks during the switch but that’s on me :slightly_smiling_face:

&lt;levity&gt;I blame the `manipulative-chimpanzee`,  it is obviously a troublemaker&lt;/levity&gt;.

<@U01EAL27T42> I benchmarked this just now. To make a fair comparison, I used the same storage for both agents - Github storage. Also, both agents are running on the same machine - my laptop. Flows:
1. This flow running on a local Kubernetes cluster: <https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/github_kubernetes_run.py> 
2. This flow running on a local agent: <https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/github_local_run.py> 
The LocalRun is consistently showing 3 sec runtime. The KubernetesRun is showing 3-4 sec.

so at this time there seem to be basically almost no difference. <@U02M50V30G4> has made some good points that you can check, but I would blame the latency to:
1. Regional distances
2. Other possible networking latency
3. Could it be you use a different storage for each? Some storage classes are faster to download than others, e.g. pulling an image from a remote registry usually takes longer

This would correspond to my uninformed expectations - that, _ceteris paribus_, local would beat clustered by the small overhead introduced by cluster operations overhead.

Also another "a bit of a stretch" scenario.. when you did the local run, did you actually shut down the cluster you were not using? (assuming same hardware)

no, the cluster was still running. But both the local agent and kubernetes agent are lightweight processes that offload execution to yet another process (local Kubernetes job vs. local subprocess) so not sure how this affects things :slightly_smiling_face: but yeah, good point, we should do it actually to ensure “ceteris paribus” :thumbsup:

same storage [S3].  real mystery. i’ll continue to investigate

*chuckles* I'm sorry, Anna &amp; Dotan- I know bugger-all about prefect (in fact struggling to learn) but I've been around distributed systems more than it might possibly be healthy, so... <https://ably.com/blog/8-fallacies-of-distributed-computing> is always close to mind.