https://prefect.io logo
g

Gregory Hunt

07/26/2023, 8:39 PM
I have 2 workers listening to the same queue, and occasionally have them both pick up the same flow and both try to run it. Is there a way to prevent this?
Bumping, Is there a way to prevent this or at least prevent it from triggering an alert?
c

Christopher Boyd

07/27/2023, 6:30 PM
Hi Gregory, Workers can both poll from the same queue - you can’t actively prevent them from both attempting to pick up the same work as that enters a whole realm of scheduling / concurrency access that would be performance decreasing - think semaphores and spin locks in the linux kernel. The workers should gracefully handle the transition (attempting to prevent state transitions from pending -> running when it’s already running on another worker), but no, you can’t actively prevent it from occurring in the first place unless you just have a single worker listening to a single pool
g

Gregory Hunt

07/27/2023, 6:32 PM
@Christopher Boyd is there an way we could prevent the alert, in my case I am using cloudrun jobs for execution, so it only lets 1 actually run at a time, but I get crashed alerts for second one because it can't start the job
c

Christopher Boyd

07/27/2023, 6:35 PM
Prevent the alert in what way? To the perspective of the worker it’s valid error - it can’t schedule work as it was successfully deterred by the API (since the state was already in running). Perhaps there is an opportunity for enhancement here maybe that attempting to schedule / transition a flow run that is already in progress returns a warning or something instead of an error (but still refuses to take action)?
g

Gregory Hunt

07/27/2023, 6:36 PM
I agree, I want to errors to be something that requires action, otherwise real errors/crashes will get hidden in the noise
I was looking to see if GCP supported an idempotent style call when creating the job, so as not to get an error
c

Christopher Boyd

07/27/2023, 6:39 PM
Do you have an example of the rror you are seeing just to refresh my memory, and so I can discuss internally?
g

Gregory Hunt

07/27/2023, 6:40 PM
Copy code
Prefect flow run notification
Flow run Files from Main Query Flow/strange-kingfisher entered state Crashed at 2023-07-27T18:32:13.744538+00:00.
Flow ID: 78c64855-3bfe-4a44-89ed-e1cd92c699e7
Flow run ID: 96fa5670-d3e7-4759-a9ed-e29ab3c3011d
Flow run URL: <http://10.0.64.30:80/flow-runs/flow-run/96fa5670-d3e7-4759-a9ed-e29ab3c3011d>
State message: Flow run could not be submitted to infrastructure