Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
j
Joshua Grant
04/20/2023, 5:10 PM@Marvin are there any reasons why prefect hosted on AWS would have an agent submit the flow request twice, causing a crash?
m
Marvin
04/20/2023, 5:12 PMI apologize for the confusion earlier. It seems that there are no existing GitHub issues related to your problem. If you're still experiencing the issue, I recommend opening a new issue on the Prefect GitHub repository to seek assistance from the Prefect community and developers. Make sure to provide detailed information about your setup, agent configuration, and the issue you're facing to help them better understand and address your problem.
n
Nate
04/20/2023, 5:13 PM@Marvin can you reconsider the question, focus on the relationship between the agent and the ECS infrastructure block
:upvote: 1
:gratitude-thank-you: 1
dang, looks like were hitting some network timeouts from openai right now
@Joshua Grant can you share your crash logs?
j
Joshua Grant
04/20/2023, 5:19 PMno worries. I think it's related to RDS
Essentially the agent completes submission of the flow, the flow does something like this:
Apr 19 21:44:54.333
prefect-v2-agent
01:44:54.333 | INFO | prefect.agent - Submitting flow run '9d15021f-e169-41c7-b826-3af39c6b9ef0'
Apr 19 21:44:54.315
prefect-v2-agent
01:44:54.314 | INFO | prefect.agent - Submitting flow run '9d15021f-e169-41c7-b826-3af39c6b9ef0'01:46:00.764 | INFO | prefect.engine - Engine execution of flow run '9d15021f-e169-41c7-b826-3af39c6b9ef0' aborted by orchestrator: This run cannot transition to the RUNNING state from the RUNNING state.01:47:25.236 | INFO | prefect.engine - Engine execution of flow run '9d15021f-e169-41c7-b826-3af39c6b9ef0' aborted by orchestrator: This run has already terminated.n
Nate
04/20/2023, 5:25 PMhmm, who's responsible for submitting these flow runs? is it the scheduler / an automation / api call?
j
Joshua Grant
04/20/2023, 5:25 PMAPI call.
z
Zanie
04/20/2023, 5:25 PMThose timestamps for submission are really close to each other. Are you only running one agent there?
j
Joshua Grant
04/20/2023, 5:27 PMEvery flow has its own agent and work-queue.
z
Zanie
04/20/2023, 5:28 PMInteresting. Does this happen often?
j
Joshua Grant
04/20/2023, 5:29 PMNo, this has happened in our beta environment. Never happened in dev or alpha, but is a concern preventing promotion to prod. This is the only occurrence.
z
Zanie
04/20/2023, 5:29 PMHm very weird
Here’s a draft poking at a possible bug? It’d be super weird though https://github.com/PrefectHQ/prefect/pull/9280
I’m a bit confused by those logs being out of order (the submitting ones)
j
Joshua Grant
04/20/2023, 5:32 PMsorry, copied and pasted from DataDog, it's default is most recent on top
z
Zanie
04/20/2023, 5:40 PMAhh okay 👍
j
Joshua Grant
04/20/2023, 8:02 PMha, dev-ops team had replicas on the agents, believe that was causing the issue
z
Zanie
04/20/2023, 9:39 PMSounds about right 😄 we still try to guard against race conditions there but within 2ms is pretty tough