Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Have observed a significant difference in overall execution time between running a flow manually and running the same flow via a local agent on the same machine (triggered via prefect cloud). Consistently 20-25 times slower for this particular flow (lots of mapped tasks). Expected behaviour?

Hey David — Hmmm that’s a pretty big difference; are you using Prefect Cloud or are you spinning up Prefect Server locally?  There is a significant architectural difference between the two, in that every API call to Server hits the database, so between logs and states I would expect _some_ of slow down but not a 20x slowdown

Hi Chris. Using Prefect Cloud. Anticipated a bit of chatter between agent and cloud. Would like to understand better how much overhead we should expect. Just testing a denser flow with 40+ mapped tasks (prior example had 7). Runs in 10 seconds or so locally but hangs mid-way through in Prefect Cloud.

That’s really interesting and not at all expected.  Almost every API call that a Flow makes to Cloud should be essentially instantaneous.  I’ve run mapped tasks of tens of thousands of tasks before, and while there was some latency overhead at the beginning of the run nothing like 20x — what executor are you using?

Thank you Chris. Good to know. It's the local agent running on a laptop.

<@UKNSNMUE6> any pointers for us to proceed?

Hey <@U0117374SBD> - apologies for the radio silence; I think we’ll need to take a deep dive here and:

- confirm your internet connection is normal speed
- figure out where the latency comes from

For point 2, we’ll want to time the various API calls that your flow makes (in particular setting states and writing logs).  We might need to write some custom code to help you to do this, but I think the broader community would benefit from a “debug” or “diagnostic” setting on Flows also

we’ll figure it internally to see if we can open-source some of this code to make it helpful to the community

Ok great! Yea if you could share your code with us that would potentially help; I’ll get the ball rolling on the diagnostic view

Hi <@U0110TYAM9C> and <@U0117374SBD> - wanted to let you know that we believe we can consistently recreate the freezing flow run using a Local Agent; will let you know when we determine the root cause!

We think we identified the underlying issue and have a possible fix here: <https://github.com/PrefectHQ/prefect/pull/2293>

If you’d like to test it sooner rather than later, I recommend installing Prefect directly off the master branch once this is merged and trying to run your mapped flow again

Ta <@UKNSNMUE6>! Will give it a whirl. :slightly_smiling_face: