Okay so I have my container built on azure container registry, but Docker() storage refuses to connect. How do I choose the container in Azure Container registry?

I think Anna has a full example here

@Sarah Floris I spent hours debugging this so AMA 😂

But I can login via az acr login

ideally, don't use az acr login, it's useless since it's only valid for 3 hours

so I use az acr login to build my docker image and to push. Is there a better way to do that then?

I wouldn't recommend creating one Azure container registry for all your Prefect images, create a service principal as described in this section https://discourse.prefect.io/t/how-to-spin-up-a-docker-agent-on-azure-vm-a-full-walkthrough/407#authenticate-with-acr-7 then login using

docker login <http://YOUR_REGISTRY_NAME.azurecr.io|YOUR_REGISTRY_NAME.azurecr.io> -u $USER_NAME -p $PASSWORD

oi I just saw that hahah

this way you need to log in only once which is much less annoying 🙂 and it's pretty much the only way to use it reliably with Docker agent since Docker agent needs long-lived credentials to be able to always pull images

so if I were to run this on kubernetes, I would have to give it a docker agent and kubernetes agent?

Just the Kubernetes agent and then you supple the image to KubernetesRun or it will use the one from DockerStorage by default

so then would I even need the Docker storage?

Yes if that is where the Flow is contained but you can do something like Github storage + KubernetesRun with a base image for example and then Prefect will run the Github file on top on that image. But if your flow actually lives in a Docker image, then yes you need to specify that as your storage

Flow storage, packaging code dependencies into a Docker image, and handling docker container registry secrets are three different issues, and you have a lot of flexibility in how you set your deployment patterns in that regard with Prefect. Flow storage vs. docker image for your code dependencies There are many flow storage types and storage configuration is independent of your Docker image (the only exception to this is Docker storage which kind of combines those together, which may be the source of your confusion). You can set a custom image on your

KubernetesRun

but it doesn't mean that you have to use

Docker

storage - storage is only to store your Flow definition (so that Prefect can pull it at runtime) while the Docker image is used to package your code dependencies and to define your flow run execution environment (e.g. Python packages that your flow run needs such as pandas, numpy, etc.). You can set a custom Docker image you built yourself on

KubernetesRun(image="yourimage")

to define code dependencies for your flow, but for flow storage you can still use e.g.

GitHub

storage or

Azure

storage - Kevin explained that already but LMK if something is still not clear about Storage. Handling docker container registry secrets Once you built your Docker image, you need to reference the image name and tag on your

KubernetesRun

run configuration so that Prefect knows which image to use. This image defines your flow run execution environment. But your Kubernetes or Docker agent also need to have proper permissions set up to pull this image at runtime. The way you set those permissions is different for Docker and Kubernetes agents. #1 To authenticate a

DockerAgent

with ACR, you need to run:

docker login <http://YOUR_REGISTRY_NAME.azurecr.io|YOUR_REGISTRY_NAME.azurecr.io> -u $USER_NAME -p $PASSWORD

then you can start your docker agent:

prefect agent docker start --label yourlabel

#2 For

KubernetesAgent

, you need to create a Kubernetes Secret which you can create for Azure AKS as follows (note the secret name "aks" here):

kubectl create secret docker-registry aks \
--docker-server=<http://prefectdemos.azurecr.io|prefectdemos.azurecr.io> \
--docker-username=prefectdemos \
--docker-password=$SP_PASSWD

you can use the same username and password as discussed before with the service principal or use this walkthrough https://github.com/anna-geller/packaging-prefect-flows/#azure-container-registry Then to use this Kubernetes secret on your run config in your flow you can set the secret name we configured above using the argument `image_pull_secrets=["aks"]`:

with Flow(
    FLOW_NAME,
    storage=STORAGE,
    run_config=KubernetesRun(
        image="<http://prefectdemos.azurecr.io/community/flows|prefectdemos.azurecr.io/community/flows>",
        labels=["aks"],
        image_pull_secrets=["aks"],  # see README
    ),
) as flow:
    hw = hello_world()