https://prefect.io logo
Title
j

Joseph Loss

12/06/2022, 5:19 PM
deploying Azure Container Instances - I keep getting a timeout waiting on container creation. Has anyone else experienced this?
1
r

Ryan Peden

12/06/2022, 5:23 PM
I've seen it occasionally depending on which Azure region I'm using. Some seem to take much longer than others. Also, if you're using EXTRA_PIP_PACKAGES to add extra packages at container start, that can add quite a bit of startup time depending on which packages you're adding. You can try adjusting the timeout setting on the
AzureContainerInstanceJob
block, or deploying an image to a private Azure Container Registry; that would prevent ACI from needed to pull the image from Docker Hub every time it starts a container.
j

Joseph Loss

12/06/2022, 5:27 PM
no extra pip packages; using a custom image. I've been trying ACR but I keep running into an issue saying that the image registry or credential isn't correct when it trys to pull the image. I've made sure that the username and password are correct. How did you do this?
r

Ryan Peden

12/06/2022, 5:28 PM
We recently released an updated version of the
prefect-azure
package that should fix that issue. If you aren't using
0.2.3
yet, I recommend updating. After that, ACR should work 🙂
🙌 1
j

Joseph Loss

12/06/2022, 5:44 PM
if using a custom image, do I need to specify a certain entry point or command for an AzureContainerInstanceJob block?
r

Ryan Peden

12/06/2022, 5:46 PM
as long as Python and Prefect are installed, the default command for a Prefect deployment (which I believe is "python -m prefect.engine") should work, so I don't think you'll need to specify one
j

Joseph Loss

12/06/2022, 6:13 PM
getting warmer but I think I need something for entrypoint:
Error: Failed to start container 7bae5840-cc55-4d9e-a649-76da33d47f96, Error response: to create containerd task: failed to create shim task: failed to create container b52528aeb53e4a508fc2fc5294a3f3942fe68a1788897bdd72ea84c5a6dd79d3: guest RPC failure: failed to create container: failed to run runc create/exec call for container b52528aeb53e4a508fc2fc5294a3f3942fe68a1788897bdd72ea84c5a6dd79d3 with exit status 1: container_linux.go:380: starting container process caused: exec: "/opt/prefect/entrypoint.sh": stat /opt/prefect/entrypoint.sh: no such file or directory: unknown
r

Ryan Peden

12/06/2022, 6:31 PM
ah - right, it's going to try to use a default entrypoint. Does it work if you set
exec
as the entrypoint on your
AzureContainerInstanceJob
block?
j

Joseph Loss

12/06/2022, 6:44 PM
no but perhaps /bin/bash -c ? will try
r

Ryan Peden

12/06/2022, 6:58 PM
Good idea! Let me know if that worked for you.
j

Joseph Loss

12/06/2022, 7:16 PM
when using exec it just gets stuck in a restart loop
r

Ryan Peden

12/06/2022, 7:19 PM
Do the logs show anything useful? I'd suggested exec because it is what Prefect's default entrypoint script uses to run flows in the container. I'll do some testing on Azure to try and document the best way to address this
j

Joseph Loss

12/06/2022, 7:21 PM
thank you. No that's the weird part, nothing in logs or in the Events section
just tried /bin/bash -c as the entrypoint:
Error: Failed to start container af25814a-9fe2-4f02-8886-15d80462961e, Error response: to create containerd task: failed to create shim task: failed to create container e271e513e9c6387e035c9a787805339040ef3e7bd66f67535ce0e435c11e8ec3: guest RPC failure: failed to create container: failed to run runc create/exec call for container e271e513e9c6387e035c9a787805339040ef3e7bd66f67535ce0e435c11e8ec3 with exit status 1: container_linux.go:380: starting container process caused: exec: "/bin/bash -c": stat /bin/bash -c: no such file or directory: unknown
r

Ryan Peden

12/06/2022, 7:24 PM
Hmm. What are you using as a base image for your image? I'll try to reproduce this as closely as I can
j

Joseph Loss

12/06/2022, 7:25 PM
ugh then passing in
/bin/bash -c python -m prefect.engine
I see this as the result in container properties: [ "/bin/bash -c python -m prefect.engine", "python", "-m", "prefect.engine" ]
it's a private multi-stage dockerfile derived from python3.9-slim-buster. I could give you the pyproject.toml ?
r

Ryan Peden

12/06/2022, 7:28 PM
That would be helpful if you can. Feel free to DM it if you'd prefer that.