So I have my NetCDF files to Zarr pipeline orchest...
# prefect-community
So I have my NetCDF files to Zarr pipeline orchestrated by Prefect to work smoothly, from firing up a temporary Fargate cluster until shutting down the EC2 instance , but I have one question I wonder
βœ… 1
When all the subflows completed, and in fact, everything is completed, the status of the main flow is still running? What should I have to look at, or it takes some time ? Thank you.
btw, ingesting few hundreds NetCDF files into Zarr within 93seconds is pretty happy results.
@Anna Geller
"When you redeploy your ECS service, both the old and new services will be running at the same time for a very short period of time (a couple of seconds). Then, the old service will be shut down. It’s likely that when the old container instance gets terminated, as a result of this transition, one flow run will remain in a Running state (it can’t finish since the underlying container is gone). In terms of your flow process, everything is fine β€” one flow run instance is still running at the same time, but it now runs in a different container."
πŸ™Œ 1
Thanks @Anna Geller much appreciated
That explains why when I use local cluster, the flows can "Completed", but when using serverless cluster, it will stuck with "Running". Thanks again, now I can explain to others within organisation.
πŸ™Œ 1
@Anna Geller I can actually fix that in my flows to use delayed shutdown request for the EC2 instance rather than immediate shutdown after finishing subflows. Thanks again
πŸ™Œ 1
πŸ’― 1