So in HA setup in AWS ECS we have 3 of each of the services for server: hasura, apollo, graphql, ui, and towel. We have in another cluster three agents running on ECS. I’ve seen a log where two agents submitted execution of the same flow run. The first two lines of the log, with the same timestamp to the second, was the “Submitted for execution: Task arnawsecs:…” from two different agents. Is there any type of locking occurring that should prevent this? Where in the code from server is this handled? I’m willing to look into it, but be nice to have a starting point of which service is responsible for handing jobs to agents and would allow for adding a check.

Hi @Joe McDonald - if the agents have identical labels then no, there's no option for this in Server; Prefect Cloud has a feature called Version Locking that prevents this sort of scenario using incrementing versions of a flow run. There's a ticket open for this in Server but I don't think it's been picked up yet.

Thanks @nicholas, for a better understanding, what are the states of a flow? Is it Scheduled->Submitted->Running ? When do states get updated, after an agent queries for a flow run or after it sends back that is has submitted the job?

Those are the states, yes: Scheduled => Submitted => Running => Finished; the submitted state gets updated by the agent after it has successfully submitted your run for execution