So I have my NetCDF files to Zarr pipeline orchestrated by Prefect to work smoothly, from firing up a temporary Fargate cluster until shutting down the EC2 instance , but I have one question I wonder
β 1
When all the subflows completed, and in fact, everything is completed, the status of the main flow is still running? What should I have to look at, or it takes some time ? Thank you.
btw, ingesting few hundreds NetCDF files into Zarr within 93seconds is pretty happy results.
"When you redeploy your ECS service, both the old and new services will be running at the same time for a very short period of time (a couple of seconds). Then, the old service will be shut down. Itβs likely that when the old container instance gets terminated, as a result of this transition, one flow run will remain in a Running state (it canβt finish since the underlying container is gone). In terms of your flow process, everything is fine β one flow run instance is still running at the same time, but it now runs in a different container."
π 1
v
Viet Nguyen
08/06/2022, 8:16 AM
Thanks @Anna Geller much appreciated
That explains why when I use local cluster, the flows can "Completed", but when using serverless cluster, it will stuck with "Running". Thanks again, now I can explain to others within organisation.
π 1
@Anna Geller I can actually fix that in my flows to use delayed shutdown request for the EC2 instance rather than immediate shutdown after finishing subflows. Thanks again