Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

image.png

Hi all! Thanks for a great tool and community.
*Summary:* I'm trying to get a deployment up and running in Azure Container Instances but Prefect Cloud keeps stopping my container after a few minutes. Do you have any ideas of what could be happening?

I'm running a flow using deployments in Azure Container Instances. I manage to successfully trigger the job and the container starts running and processing the tasks correctly. After a while, for no apparent reason, the agent receives the instruction from prefect (running on Cloud) to delete the container:
```16:44:27.874 | INFO    | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Preparing to run command '/opt/prefect/entrypoint.sh python -m prefect.engine' in container 'a53fd966-c635-4605-a8ab-4a3f487a7463' (prefecthq/prefect:2.6.7-python3.9)...
16:44:27.875 | INFO    | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Waiting for container creation...
16:45:37.024 | INFO    | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Running command...
16:45:37.025 | INFO    | prefect.agent - Completed submission of flow run '6bb0fc44-197e-4bfd-846e-877f5dbbf0b8'
16:48:43.454 | INFO    | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Completed command run.
16:48:43.455 | INFO    | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Deleting container...```
In the meantime the Orion UI gets frozen in 'running' state and there are no log lines indicating any error or crash.

I have no help to offer, but I'm having a similar issue in Kubernetes where my jobs are getting OOMKilled (because of limits I have in place in the deployment) but no errors are being reported. Prefect tries to re-run the job but immediately completes because it's already in a RUNNING state and the run gets indefinitely stuck in the UI as running.

Hi Guillermo! The `AzureContainerInstanceJob` checks the status of your container every 5 seconds and based on the output, it looks like the container is getting deleted because it has stopped running.

Prefect isn't stopping the stopping the container; it actually doesn't have a way to stop the container mid-run (though it will soon). So something else is causing the flow run to stop prematurely.

Enabling log streaming on your `AzureContainerInstanceJob` block (if it's not enabled already) might reveal more information about what is going wrong.

Thanks a lot! I'll look into the logs in the container and try to see what might be happenning. I'll keep you posted with my results.

On the other hand, it seems a bit odd that the flow will maintain a "Running" state in this case, what's the mechanism for detecting if it's crashed, running, etc?