https://prefect.io logo
Title
d

Dave

11/07/2020, 3:03 PM
Hey guys, Have someone experienced that one flow runs twice in the same execution? Due to a multiple local agent configured at the exact same time regardless of you trigger it manually or its scheduled. Duplicate runs / duplicate executions. (Check Thread)
j

josh

11/09/2020, 3:50 PM
Hi @Dave are you using a schedule with your flow by any chance? Someone recently opened a report of something like this happening w/ server and it may be related 🙂 https://github.com/PrefectHQ/prefect/issues/3629
d

Dave

11/09/2020, 3:56 PM
Hello, yes I've a scheduled configured on my flow:
..
schedule = IntervalSchedule(
    start_date=pendulum.now(),
    interval=pendulum.duration(minutes=15)

with Flow("check_position", schedule, state_handlers=[state_handler]) as flow:
...
But it also happens on non-scheduled flows. Updated: In my current setup I've two local agent's with the same labels and my environment is as follows:
prefect diagnostics:
{
  "config_overrides": {},
  "env_vars": [],
  "system_information": {
    "platform": "Linux-5.4.0-1031-azure-x86_64-with-glibc2.29",
    "prefect_backend": "server",
    "prefect_version": "0.13.14",
    "python_version": "3.8.5"
  }
}
When the Agent's have the same labels it both triggers my Flow:
7 November 2020,03:28:38 	Agent01	INFO	Submitted for execution: PID: 1161
7 November 2020,03:28:38 	Agent02	INFO	Submitted for execution: PID: 1147
7 November 2020,03:28:39 	prefect.CloudFlowRunner	INFO	Beginning Flow run for 'post_msg_slack'
7 November 2020,03:28:39 	prefect.CloudFlowRunner	INFO	Beginning Flow run for 'post_msg_slack'
7 November 2020,03:28:39 	prefect.CloudTaskRunner	INFO	Task 'SlackMsg': Starting task run...
7 November 2020,03:28:39 	prefect.CloudTaskRunner	INFO	Task 'SlackMsg': Starting task run...
7 November 2020,03:28:39 	prefect.CloudTaskRunner	INFO	Task 'SlackMsg': Finished task run for task with final state: 'Success'
7 November 2020,03:28:39 	prefect.CloudTaskRunner	INFO	Task 'SlackMsg': Finished task run for task with final state: 'Success'
7 November 2020,03:28:40 	prefect.CloudTaskRunner	INFO	Task 'post': Starting task run...
7 November 2020,03:28:40 	prefect.CloudTaskRunner	INFO	Task 'post': Starting task run...
7 November 2020,03:28:40 	prefect	INFO	Req: URL|[{'via': 'slack', 'to': ['user', 'dmo'], 'text': 'Hello from UI!'}]
7 November 2020,03:28:41 	prefect	INFO	Req: URL|[{'via': 'slack', 'to': ['user', 'dmo'], 'text': 'Hello from UI!'}]
7 November 2020,03:28:41 	prefect.CloudTaskRunner	INFO	Task 'post': Finished task run for task with final state: 'Success'
7 November 2020,03:28:41 	prefect.CloudFlowRunner	INFO	Flow run SUCCESS: all reference tasks succeeded
7 November 2020,03:28:41 	prefect.CloudTaskRunner	INFO	Task 'post': Finished task run for task with final state: 'Success'
7 November 2020,03:28:41 	prefect.CloudFlowRunner	INFO	Flow run SUCCESS: all reference tasks succeeded
Just to clarify I'm registering the flows within the same server and with the same Prefect version as Server and UI and they both have the same version. If I then don't configure my agent's at the exact same time it only trigger it once, meaning if I wait one second starting the other agent the above doesn't happen. After a while it's then possible for my agents to be querying the server at the same time for flows, where it again trigger it twice. For me it's not clear if this is expected behaviour since I couldn't find anything in the docs about it.
Answer: Prefect Server does not implement any lock on states so if you run horizontally scale agents there is a chance they submit the same run twice. Prefect Cloud uses a global lock that prevents this behavior.