deploying Azure Container Instances - I keep getting a timeout waiting on container creation. Has anyone else experienced this?

I've seen it occasionally depending on which Azure region I'm using. Some seem to take much longer than others. Also, if you're using EXTRA_PIP_PACKAGES to add extra packages at container start, that can add quite a bit of startup time depending on which packages you're adding. You can try adjusting the timeout setting on the

AzureContainerInstanceJob

block, or deploying an image to a private Azure Container Registry; that would prevent ACI from needed to pull the image from Docker Hub every time it starts a container.

no extra pip packages; using a custom image. I've been trying ACR but I keep running into an issue saying that the image registry or credential isn't correct when it trys to pull the image. I've made sure that the username and password are correct. How did you do this?

We recently released an updated version of the

prefect-azure

package that should fix that issue. If you aren't using

0.2.3

yet, I recommend updating. After that, ACR should work 🙂

if using a custom image, do I need to specify a certain entry point or command for an AzureContainerInstanceJob block?

as long as Python and Prefect are installed, the default command for a Prefect deployment (which I believe is "python -m prefect.engine") should work, so I don't think you'll need to specify one

getting warmer but I think I need something for entrypoint:

Error: Failed to start container 7bae5840-cc55-4d9e-a649-76da33d47f96, Error response: to create containerd task: failed to create shim task: failed to create container b52528aeb53e4a508fc2fc5294a3f3942fe68a1788897bdd72ea84c5a6dd79d3: guest RPC failure: failed to create container: failed to run runc create/exec call for container b52528aeb53e4a508fc2fc5294a3f3942fe68a1788897bdd72ea84c5a6dd79d3 with exit status 1: container_linux.go:380: starting container process caused: exec: "/opt/prefect/entrypoint.sh": stat /opt/prefect/entrypoint.sh: no such file or directory: unknown

ah - right, it's going to try to use a default entrypoint. Does it work if you set

exec

as the entrypoint on your

AzureContainerInstanceJob

block?

no but perhaps /bin/bash -c ? will try

Good idea! Let me know if that worked for you.

when using exec it just gets stuck in a restart loop

Do the logs show anything useful? I'd suggested exec because it is what Prefect's default entrypoint script uses to run flows in the container. I'll do some testing on Azure to try and document the best way to address this

thank you. No that's the weird part, nothing in logs or in the Events section

Hmm. What are you using as a base image for your image? I'll try to reproduce this as closely as I can

ugh then passing in

/bin/bash -c python -m prefect.engine

I see this as the result in container properties: [ "/bin/bash -c python -m prefect.engine", "python", "-m", "prefect.engine" ]

That would be helpful if you can. Feel free to DM it if you'd prefer that.