Is it possible for a scheduled flow to run on multiple agents at the same time, if all labels match? Is this expected or is it a bug? We are using Cloud. https://prefect-community.slack.com/archives/CL09KU1K7/p1595309512466100

Hi @Klemen Strojan, there are situations in which it is possible and even desirable, so we do not explicitly prevent it, but it’s unusual. Is it possible for you to share the flow run id?

Sure @Jeremiah, here it is:

2b2168a6-456c-42b4-ad3d-88fe5118b62e

. It is good to know that this is desirable.

Hi Klemen - it’s certainly unusual and most of the time we’d expect only one submission. The reason we don’t prevent it is that it’s possible (though again unlikely) for multiple “isntances” of a flow to need to run simultaneously — for example, if multiple tasks all request retries at slightly different future times. Coordinating that activity is a challenging distributed systems problem 😅 and sometimes results in behaviors like the one you saw. However, I’m flagging the ID for our team and we’ll see if we can identify the circumstances and if there are additional steps we can take to limit this repetition. One thing I already know is the upcoming release of Core will improve what we call “version locking” which is our way of coordinating activity among potentially multiple distrubuted agents. Those improvements plus some changes we’ll enable in Cloud will reduce the possibility of this further.

Thanks for this explanation - it makes a lot of sense and we can mitigate this behaviour w/o any hacking. We like your product very much, keep up the good work!

Thanks!