We're running all our flows via ECSRun to kick off a Fargate task to run the flow. It all works fine but what I'm seeing are a lot of Fargate zombie tasks still running after the flow has finished. I need to investigate this a little more as it may be failed flows and/or flows we cancel via the Prefect console... not sure at this stage. Has anyone else had a similar issue? (a quick search of this channel didn't reveal much). Many thanks!
k
Kevin Kho
01/04/2022, 4:24 PM
Hi @Theo Platt, are the Flows finished in the Prefect UI? Not ECS specific but I have seen Docker connections fail to close because of connections to databases that are still open. Do you think you can have something like that?
t
Theo Platt
01/04/2022, 4:27 PM
Thanks @Kevin Kho. Yes - the flows are either cancelled, successful or failed. We're going to do some more digging to see if we can replicate the issue.
k
Kevin Kho
01/04/2022, 4:28 PM
I don’t think it’s related, but just wanted to put this on your radar also
t
Theo Platt
01/04/2022, 4:41 PM
I think it may be related. We have been cancelling a lot of flows.
k
Kevin Kho
01/04/2022, 4:43 PM
But I guess it also happened for Failed flows which is why I’m not too sure. I am not too sure but I see this more on k8s rather than ECS. Could be related though.
t
Theo Platt
01/04/2022, 6:54 PM
I need to confirm the failed flows - may not be the case for them - may just be cancelled flows