👋 Hello! I am seeing some odd behavior in one of our flow runs. Essentially, mid-run, it spins up another process with the same

uuid

and

name

, and then continues to do so multiple times, until eventually one completes or the process OOM-dies. Below are the state transitions for one of the runs:

2023-09-05T18:53:31.140114+00:00 SCHEDULED Scheduled
2023-09-06T06:29:59.024936+00:00 PENDING Pending
2023-09-06T06:30:07.054389+00:00 RUNNING Running
2023-09-06T08:31:21.992053+00:00 SCHEDULED AwaitingRetry
2023-09-06T08:31:30.726178+00:00 PENDING Pending
2023-09-06T08:31:37.152022+00:00 RUNNING Running
2023-09-06T08:34:21.507672+00:00 SCHEDULED AwaitingRetry
2023-09-06T08:34:24.847413+00:00 PENDING Pending
2023-09-06T08:34:31.339831+00:00 RUNNING Running
2023-09-06T08:40:25.379897+00:00 SCHEDULED AwaitingRetry
2023-09-06T08:40:33.138487+00:00 PENDING Pending
2023-09-06T08:40:40.528714+00:00 RUNNING Running
2023-09-06T08:43:21.708016+00:00 SCHEDULED AwaitingRetry
...

The logs on our end also show that during the run, other flows will spin up with:

Sep 6 02:28:29 317c1000d734 X info 08:28:00.265 | INFO    | Flow run 'positive-salmon' - Syncing _ from 2023-07-19 to 2023-07-28
Sep 6 02:31:54 317c1000d734 X info 08:31:31.118 | INFO    | prefect.infrastructure.process - Opening process 'positive-salmon'...
Sep 6 02:31:54 317c1000d734 X info 08:31:31.119 | DEBUG   | prefect.infrastructure.process - Process 'positive-salmon' running command: /usr/local/bin/python -m prefect.engine in /tmp/tmpob1m856jprefect
Sep 6 02:31:54 317c1000d734 X info 08:31:34.899 | INFO    | Flow run 'positive-salmon' - Downloading flow code from storage at None
Sep 6 02:31:54 317c1000d734 X info 08:31:35.985 | DEBUG   | Flow run 'positive-salmon' - Importing flow code from '[filepath]:run'
Sep 6 02:31:54 317c1000d734 X info 08:31:36.903 | DEBUG   | Flow run 'positive-salmon' - Starting 'ConcurrentTaskRunner'; submitted tasks will be run concurrently...
Sep 6 02:31:54 317c1000d734 X info 08:31:37.534 | DEBUG   | Flow run 'positive-salmon' - Executing flow 'sync-*' for flow run 'positive-salmon'...
Sep 6 02:31:54 317c1000d734 X info 08:31:37.536 | DEBUG   | Flow run 'positive-salmon' - Beginning execution...
Sep 6 02:31:54 317c1000d734 X info 08:31:38.628 | INFO    | Flow run 'positive-salmon' - Starting sync
Sep 6 02:31:54 317c1000d734 X info 08:31:38.631 | INFO    | Flow run 'positive-salmon' - Started
Sep 6 02:31:54 317c1000d734 X info 08:31:38.633 | INFO    | Flow run 'positive-salmon' - Syncing _ from 2023-08-28 to 2023-09-06
...
Sep 6 03:49:20 317c1000d734 X info 09:48:40.992 | INFO    | Flow run 'positive-salmon' - Syncing _ from 2023-08-18 to 2023-08-27 [from one of the started ones]
Sep 6 03:50:54 317c1000d734 X info 09:50:34.072 | INFO    | Flow run 'positive-salmon' - Syncing _ from 2023-06-29 to 2023-07-08 [likely from the original process]

And then go on to show no crashes, and show evidence of multiple "flows" running at the same time. The prefect UI however only shows 1 flow ran -- and the WorkQueue has a concurrency limit of 1 Any ideas here?