A few more questions that could help us figure out what’s up with your flow: • What infrastructure is being used for the run? • Are retries configured for the run? • Are any automations configured for the run? • Is the run managed by a worker or an agent? • Is it a subflow run? • Was the run trigged by

run_deployment

or a schedule? • Are you using Cloud or OSS? • If Cloud, provide workspace and flow run ids If using OSS, state transitions can be retrieved with the client with this script, and that information could be helpful:

# usage: python <file>.py <FLOW_RUN_ID>
from prefect import get_client

async def main(flow_run_id):
    async with get_client() as client:
        states = await client.read_flow_run_states(flow_run_id)
        for state in states:
            print(state.timestamp, state.type.name, state.name)


import asyncio
import sys

asyncio.run(main(sys.argv[1]))

Can you get us that?

We do have multiple agents consuming this queue.

But all logs come from same agents so it seems.

Yes it is even taking the result from the second run from agent cache.

Finished in state Cached(type=COMPLETED)

We’re interested in the infrastructure management logs on the agent

like is

Submitting flow run <id>

displayed on one agent twice or both agents once?

We have 2 pods with orion server, might it be an issue?

We definitely avoid this with a single agent by storing submitting runs in a local set

Do you need anything more from me? Also is it more likely Orion or Agent issue? Because this might be a good reason to update from 2.7.10

I haven’t seen any recent reports of this, it’s possible it’s been fixed you’re pretty far behind on versions

I have one more concern going back to initial question HERE. When running we get more than one run in flow that has 0 retries.

So taskrun has retry with backoff but flow does not. why is that the flow retires itself?

@Jeff Hale Can we continue this issue in this topic? Or should I create separate since those 2 issues are probably different

1. Running such flows with run_deployment 2. Infra: K8S job 3. Prefect 2.10.9 Flow has one task inside and task has retry with backoff but flow itself is run multiple times. Is it because of job being retried by cluster?

I think it is it because I see backoffLimit: 6 in the manifest. Sorry for the hassle

And yes the backoffLimit is the cause of the other issue. If you flow runs are on 2.10.3+ they won’t back off unless the flow run crashes. We’re also going to change the default backoff limit.