Hello fine people, I'm seeing an odd state transition fairly reguarly (at least once a day), we have...

Alex Hornbake

09/19/2023, 7:00 AM

Hello fine people, I'm seeing an odd state transition fairly reguarly (at least once a day), we have a false negative "crashed" flow run, where the state transitions from:

Scheduled → Pending → Crashed → Running → Completed

For now, we've adjusted our alerting to only trigger if a flow "stays in {state} for 5 minutes", but it would be nice to be able to alert when a flow "enters {state}". lmk if there is a better forum for this bug report.

Will Raphaelson

09/19/2023, 10:00 PM

Hey Alex, that false positive crash state is interesting. Can you tell me a bit about your infra and job setup?

Alex Hornbake

09/20/2023, 5:53 AM

We're using prefect-aws. These are ECSTasks running on Fargate

Alex Hornbake

09/20/2023, 5:59 AM

We're using prefect cloud.

Alex Hornbake

09/20/2023, 6:08 AM

Could this be a timeout issue? https://prefecthq.github.io/prefect-aws/ecs/#prefect_aws.ecs.ECSTask.task_start_timeout_seconds The default is 120 seconds, and the task transitions to "running" more than 120 seconds after it entered the "pending" state.

Alex Hornbake

09/20/2023, 6:15 AM

I might be answering my own question... prefect duck I'm guessing that is possible if the flow run reports "RUNNING" after the agent reports the run as "CRASHED". It might be helpful if the Event log in the UI indicated: • Who reported the state change (agent or task) • Why the state changed (likely due to timeout in this case)

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.