Guillermo Galan
11/21/2022, 4:02 PM16:44:27.874 | INFO | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Preparing to run command '/opt/prefect/entrypoint.sh python -m prefect.engine' in container 'a53fd966-c635-4605-a8ab-4a3f487a7463' (prefecthq/prefect:2.6.7-python3.9)...
16:44:27.875 | INFO | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Waiting for container creation...
16:45:37.024 | INFO | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Running command...
16:45:37.025 | INFO | prefect.agent - Completed submission of flow run '6bb0fc44-197e-4bfd-846e-877f5dbbf0b8'
16:48:43.454 | INFO | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Completed command run.
16:48:43.455 | INFO | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Deleting container...
In the meantime the Orion UI gets frozen in 'running' state and there are no log lines indicating any error or crash.Sam Cook
11/21/2022, 4:27 PMRyan Peden
11/21/2022, 5:41 PMAzureContainerInstanceJob
checks the status of your container every 5 seconds and based on the output, it looks like the container is getting deleted because it has stopped running.
Prefect isn't stopping the stopping the container; it actually doesn't have a way to stop the container mid-run (though it will soon). So something else is causing the flow run to stop prematurely.
Enabling log streaming on your AzureContainerInstanceJob
block (if it's not enabled already) might reveal more information about what is going wrong.Guillermo Galan
11/22/2022, 10:13 AM