https://prefect.io logo
Title
d

David Haines

04/05/2020, 6:24 AM
Have observed a significant difference in overall execution time between running a flow manually and running the same flow via a local agent on the same machine (triggered via prefect cloud). Consistently 20-25 times slower for this particular flow (lots of mapped tasks). Expected behaviour?
c

Chris White

04/05/2020, 5:43 PM
Hey David — Hmmm that’s a pretty big difference; are you using Prefect Cloud or are you spinning up Prefect Server locally? There is a significant architectural difference between the two, in that every API call to Server hits the database, so between logs and states I would expect some of slow down but not a 20x slowdown
d

David Haines

04/05/2020, 8:11 PM
Hi Chris. Using Prefect Cloud. Anticipated a bit of chatter between agent and cloud. Would like to understand better how much overhead we should expect. Just testing a denser flow with 40+ mapped tasks (prior example had 7). Runs in 10 seconds or so locally but hangs mid-way through in Prefect Cloud.
c

Chris White

04/05/2020, 10:09 PM
That’s really interesting and not at all expected. Almost every API call that a Flow makes to Cloud should be essentially instantaneous. I’ve run mapped tasks of tens of thousands of tasks before, and while there was some latency overhead at the beginning of the run nothing like 20x — what executor are you using?
d

David Haines

04/06/2020, 3:42 AM
Thank you Chris. Good to know. It's the local agent running on a laptop.
s

Shankar Vasudevan

04/06/2020, 7:00 PM
@Chris White any pointers for us to proceed?
c

Chris White

04/06/2020, 7:02 PM
Hey @Shankar Vasudevan - apologies for the radio silence; I think we’ll need to take a deep dive here and: - confirm your internet connection is normal speed - figure out where the latency comes from For point 2, we’ll want to time the various API calls that your flow makes (in particular setting states and writing logs). We might need to write some custom code to help you to do this, but I think the broader community would benefit from a “debug” or “diagnostic” setting on Flows also
s

Shankar Vasudevan

04/06/2020, 7:03 PM
that would be brilliant.
we’ll figure it internally to see if we can open-source some of this code to make it helpful to the community
c

Chris White

04/06/2020, 7:05 PM
Ok great! Yea if you could share your code with us that would potentially help; I’ll get the ball rolling on the diagnostic view
Hi @David Haines and @Shankar Vasudevan - wanted to let you know that we believe we can consistently recreate the freezing flow run using a Local Agent; will let you know when we determine the root cause!
s

Shankar Vasudevan

04/07/2020, 6:52 PM
That’s positive 💪🏻
c

Chris White

04/07/2020, 8:23 PM
We think we identified the underlying issue and have a possible fix here: https://github.com/PrefectHQ/prefect/pull/2293 If you’d like to test it sooner rather than later, I recommend installing Prefect directly off the master branch once this is merged and trying to run your mapped flow again
d

David Haines

04/07/2020, 9:04 PM
Ta @Chris White! Will give it a whirl. 🙂
💯 1