Hey!
We've encountered a bug in the Kubernetes work pool.
When running a flow that has a task with a task run concurrency limit,
if that flow receives an OOM kill from Kubernetes, it doesn't report anything to the Prefect server.
Which means that the flow remains in a running state,
and its task run concurrency limit stays locked which prevents my other tasks from starting.
I would love Prefect to somehow detect these kind of crashes (maybe a sidecar / watchdog?) and report them back.
Thanks!
i
Imre Kerr
09/17/2024, 9:48 AM
We've seen something similar on ECS Fargate (we think. Currently investigating.) Interested to hear what Prefect has to say.
🧐 1
b
Bil Tal
09/22/2024, 3:43 PM
@Alexander Azzam@Nate Any insight on this?
s
Samuel Hinton
12/04/2024, 9:40 AM
Hey @Bil Tal, did you ever resolve this? We're having the same issue with tag based concurrency
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.