Have observed a significant difference in overall execution time between running a flow manually and running the same flow via a local agent on the same machine (triggered via prefect cloud). Consistently 20-25 times slower for this particular flow (lots of mapped tasks). Expected behaviour?

Hey David — Hmmm that’s a pretty big difference; are you using Prefect Cloud or are you spinning up Prefect Server locally? There is a significant architectural difference between the two, in that every API call to Server hits the database, so between logs and states I would expect some of slow down but not a 20x slowdown

Hi Chris. Using Prefect Cloud. Anticipated a bit of chatter between agent and cloud. Would like to understand better how much overhead we should expect. Just testing a denser flow with 40+ mapped tasks (prior example had 7). Runs in 10 seconds or so locally but hangs mid-way through in Prefect Cloud.

That’s really interesting and not at all expected. Almost every API call that a Flow makes to Cloud should be essentially instantaneous. I’ve run mapped tasks of tens of thousands of tasks before, and while there was some latency overhead at the beginning of the run nothing like 20x — what executor are you using?

Thank you Chris. Good to know. It's the local agent running on a laptop.

@Chris White any pointers for us to proceed?

Hey @Shankar Vasudevan - apologies for the radio silence; I think we’ll need to take a deep dive here and: - confirm your internet connection is normal speed - figure out where the latency comes from For point 2, we’ll want to time the various API calls that your flow makes (in particular setting states and writing logs). We might need to write some custom code to help you to do this, but I think the broader community would benefit from a “debug” or “diagnostic” setting on Flows also

that would be brilliant.

Ok great! Yea if you could share your code with us that would potentially help; I’ll get the ball rolling on the diagnostic view

That’s positive 💪🏻

We think we identified the underlying issue and have a possible fix here: https://github.com/PrefectHQ/prefect/pull/2293 If you’d like to test it sooner rather than later, I recommend installing Prefect directly off the master branch once this is merged and trying to run your mapped flow again

Ta @Chris White! Will give it a whirl. 🙂