<@ULVA73B9P> are there any reasons why prefect hosted on AWS would have an agent submit the flow req...

Joshua Grant

04/20/2023, 5:10 PM

@Marvin are there any reasons why prefect hosted on AWS would have an agent submit the flow request twice, causing a crash?

Marvin

04/20/2023, 5:12 PM

I apologize for the confusion earlier. It seems that there are no existing GitHub issues related to your problem. If you're still experiencing the issue, I recommend opening a new issue on the Prefect GitHub repository to seek assistance from the Prefect community and developers. Make sure to provide detailed information about your setup, agent configuration, and the issue you're facing to help them better understand and address your problem.

Nate

04/20/2023, 5:13 PM

@Marvin can you reconsider the question, focus on the relationship between the agent and the ECS infrastructure block

upvote 1

gratitude thank you 1

Nate

04/20/2023, 5:18 PM

dang, looks like were hitting some network timeouts from openai right now

Nate

04/20/2023, 5:19 PM

@Joshua Grant can you share your crash logs?

Joshua Grant

04/20/2023, 5:19 PM

no worries. I think it's related to RDS

Joshua Grant

04/20/2023, 5:23 PM

Essentially the agent completes submission of the flow, the flow does something like this:

Copy code

Apr 19 21:44:54.333
prefect-v2-agent
01:44:54.333 | INFO | prefect.agent - Submitting flow run '9d15021f-e169-41c7-b826-3af39c6b9ef0'

Apr 19 21:44:54.315
prefect-v2-agent
01:44:54.314 | INFO | prefect.agent - Submitting flow run '9d15021f-e169-41c7-b826-3af39c6b9ef0'

it submits the flow runs twice at different times. This has only happened once. Both the flow complains:

Copy code

01:46:00.764 | INFO    | prefect.engine - Engine execution of flow run '9d15021f-e169-41c7-b826-3af39c6b9ef0' aborted by orchestrator: This run cannot transition to the RUNNING state from the RUNNING state.

Then finally dies with both running flows complaining:

Copy code

01:47:25.236 | INFO    | prefect.engine - Engine execution of flow run '9d15021f-e169-41c7-b826-3af39c6b9ef0' aborted by orchestrator: This run has already terminated.

Nate

04/20/2023, 5:25 PM

hmm, who's responsible for submitting these flow runs? is it the scheduler / an automation / api call?

Joshua Grant

04/20/2023, 5:25 PM

API call.

Zanie

04/20/2023, 5:25 PM

Those timestamps for submission are really close to each other. Are you only running one agent there?

Joshua Grant

04/20/2023, 5:27 PM

Every flow has its own agent and work-queue.

Zanie

04/20/2023, 5:28 PM

Interesting. Does this happen often?

Joshua Grant

04/20/2023, 5:29 PM

No, this has happened in our beta environment. Never happened in dev or alpha, but is a concern preventing promotion to prod. This is the only occurrence.

Zanie

04/20/2023, 5:29 PM

Hm very weird

Zanie

04/20/2023, 5:30 PM

Here’s a draft poking at a possible bug? It’d be super weird though https://github.com/PrefectHQ/prefect/pull/9280

Zanie

04/20/2023, 5:31 PM

I’m a bit confused by those logs being out of order (the submitting ones)

Joshua Grant

04/20/2023, 5:32 PM

sorry, copied and pasted from DataDog, it's default is most recent on top

Zanie

04/20/2023, 5:40 PM

Ahh okay 👍

Joshua Grant

04/20/2023, 8:02 PM

ha, dev-ops team had replicas on the agents, believe that was causing the issue

Zanie

04/20/2023, 9:39 PM

Sounds about right 😄 we still try to guard against race conditions there but within 2ms is pretty tough

6 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.