Hey all, my team has been using ECSTask block infr...
# prefect-awsm
Mike Grabbe
01/10/2023, 3:33 PMHey all, my team has been using ECSTask block infrastructure for a couple months, and we've noticed that flows will occasionally fail with an error similar to
Submission failed. RuntimeError: Timed out after 240.94525742530823s while watching task for status {until_status or 'STOPPED'}✅ 1
k
Kalise Richmond
01/10/2023, 3:59 PMHey @Mike Grabbe, do you happen to have debug logs from the agent that is running this? Which version of Prefect are you using?
m
Mike Grabbe
01/10/2023, 4:00 PMIm on latest version, 2.7.7
Mike Grabbe
01/10/2023, 4:00 PMI'll try to dig up the logs
z
Zanie
01/10/2023, 4:16 PM
Retries are for errors in the flow itself right now, this is an error outside the flow (i.e. in infrastructure management). We have a goal to add separate retry settings for that.
Zanie
01/10/2023, 4:17 PM
It sounds like you want to increase https://prefecthq.github.io/prefect-aws/ecs/#prefect_aws.ecs.ECSTask.task_start_timeout_seconds ? Although the default is 120s so it sounds like you already have perhaps?
m
Mike Grabbe
01/10/2023, 4:32 PMRight, I did. My guess is that increasing the timeout any more wont do any good. These are proably ECS task launch failures
Mike Grabbe
01/10/2023, 4:33 PMOK so the scope of the retry configuration for now is limited to the task or the flow, but its not yet available at the infra level
z
Zanie
01/10/2023, 4:40 PM
Yep!
Zanie
01/10/2023, 4:41 PM
m
Mike Grabbe
01/10/2023, 4:49 PMThanks @Zanie, glad to see this is already on the roadmap
Mike Grabbe
01/11/2023, 6:21 PMFollowing up on this, @Kalise Richmond: I added the stack trace from the agent logs into the github issue linked directly above.
👍 1
3 Views