https://prefect.io logo
Title
k

Klemen Strojan

07/23/2020, 12:18 PM
Is it possible for a scheduled flow to run on multiple agents at the same time, if all labels match? Is this expected or is it a bug? We are using Cloud. https://prefect-community.slack.com/archives/CL09KU1K7/p1595309512466100
j

Jeremiah

07/23/2020, 12:21 PM
Hi @Klemen Strojan, there are situations in which it is possible and even desirable, so we do not explicitly prevent it, but it’s unusual. Is it possible for you to share the flow run id?
k

Klemen Strojan

07/23/2020, 12:30 PM
Sure @Jeremiah, here it is:
2b2168a6-456c-42b4-ad3d-88fe5118b62e
. It is good to know that this is desirable.
Is it ok to say that for a “normal” flow, flow labels should point to a single agent?
j

Jeremiah

07/23/2020, 12:47 PM
Hi Klemen - it’s certainly unusual and most of the time we’d expect only one submission. The reason we don’t prevent it is that it’s possible (though again unlikely) for multiple “isntances” of a flow to need to run simultaneously — for example, if multiple tasks all request retries at slightly different future times. Coordinating that activity is a challenging distributed systems problem 😅 and sometimes results in behaviors like the one you saw. However, I’m flagging the ID for our team and we’ll see if we can identify the circumstances and if there are additional steps we can take to limit this repetition. One thing I already know is the upcoming release of Core will improve what we call “version locking” which is our way of coordinating activity among potentially multiple distrubuted agents. Those improvements plus some changes we’ll enable in Cloud will reduce the possibility of this further.
For the moment, I think this is a symptom of multiple agents with overlapping labels that we’ll seek to mitigate with the improvements I mentioned.
k

Klemen Strojan

07/23/2020, 1:05 PM
Thanks for this explanation - it makes a lot of sense and we can mitigate this behaviour w/o any hacking. We like your product very much, keep up the good work!
j

Jeremiah

07/23/2020, 1:19 PM
Thanks!