I am noticing issues with ECS and our push work po...
# ask-communityc
Chandler
02/28/2025, 6:58 PMI am noticing issues with ECS and our push work pools as well surrounding this with sudden high CPU usage. Is this still happening today?
Chandler
02/28/2025, 6:59 PMAll of our jobs have suddenly went into LATE status as of 8:00AM MTN time today.
j
Jake Kaplan
02/28/2025, 7:28 PMhey, nothing that I am aware of right now. Can you share workspace id and work pool id?
c
Chandler
02/28/2025, 7:38 PMWorkspace ID
f375b1eb-767e-43c6-abe2-3c1d1043706a
Workpool ID:
18784370-635d-49e9-b1cb-d048f829dadf
j
Jake Kaplan
02/28/2025, 7:40 PMJust to make sure I have the right one, that work pool does not appear to be a push pool
Jake Kaplan
02/28/2025, 7:44 PMI do see heartbeats from that worker stop a few hours ago? Is that worker still running? If it's stopped for some reason that would explain the late flow runs as there is nothing to pick them up
c
Chandler
02/28/2025, 8:35 PMIt appears to still be running, but the tasks are failing
j
Jake Kaplan
02/28/2025, 8:41 PMSorry, when you say tasks you mean there are errors on the worker itself? Or prefect tasks?
c
Chandler
02/28/2025, 8:42 PMCorrect, in ECS it seems we are getting stop errors for some reason but the tasks are populating. It just can’t grab the image it seems?
j
Jake Kaplan
02/28/2025, 8:47 PMGotcha. Can at least definitely confirm theres no systemic issue with Prefect Cloud. It sounds like it's an issue between ECS <--> and your image repository possibly?
Assuming your image tag is valid, I've seen transient network issues before or things like dockerhub rate limits? (not sure if you're using dockerhub). I may not be able to help so much here, but happy to take a look if you have an error to share?
c
Chandler
02/28/2025, 8:48 PMThis was mainly just to ensure it is an us problem, I’ll need to continue investigating on our end.
👍 1
Chandler
02/28/2025, 8:48 PMThanks for your help!
j
Jake Kaplan
02/28/2025, 8:48 PMno problem, sounds good! best of luck 🤞
c
Chandler
02/28/2025, 9:02 PMIt seems the orchestrator, the work pool manager is failing for us.
Chandler
02/28/2025, 9:05 PMHere is the log
j
Jake Kaplan
02/28/2025, 9:19 PMThis is from the worker? it's a little hard to read the log to understand what the actual error is/where it's occurring
c
Chandler
02/28/2025, 9:19 PMHa, yeah its just as confusing to us
😅 1
Chandler
02/28/2025, 9:19 PMSo a little context.
j
Jake Kaplan
02/28/2025, 9:19 PMare you able to get a traceback?
c
Chandler
02/28/2025, 9:21 PMThere is only one task in ecs that keeps rotating, it starts and dies every threeish minutes
j
Jake Kaplan
02/28/2025, 9:22 PMIf this started happening out of "nowhere" it's possible it's a dependency issue? are all your packages pinned?
c
Chandler
02/28/2025, 9:22 PMAll of our packages are pinned yes
Chandler
02/28/2025, 9:23 PMIts something with pydantic
Chandler
02/28/2025, 9:23 PMfrom prefect specifically I believe
Chandler
02/28/2025, 9:23 PMonce it hits that point in the log, error traces start to show up
j
Jake Kaplan
02/28/2025, 9:27 PMa little hard to tell where in the worker process that is, but is that just after running
prefect worker start ...c
Chandler
02/28/2025, 9:54 PMIt was due to the prefect image
Chandler
02/28/2025, 9:55 PMwe upgraded to latest and that fixed it
Chandler
02/28/2025, 9:55 PMprefect-recipesj
Jake Kaplan
02/28/2025, 9:58 PMoh great! sounds like maybe some sort of conflict versions? I saw in that log it looked like you were installing the latest prefect I thought
c
Chandler
02/28/2025, 9:58 PMYeah, 3.0.1 -> 3.2.9 and that fixed it
🔥 2
Chandler
02/28/2025, 9:59 PMbut like.. it was pinned lol. I guess I am confused how that would happen
Chandler
02/28/2025, 10:00 PMWhy would there be dependency changes on a pinned package from months ago
j
Jake Kaplan
02/28/2025, 10:00 PM3.2.9b
Bianca Hoch
02/28/2025, 10:25 PMHey Chandler! The newest home for prefect examples can be found here.
👀 1
2 Views