Sarah Floris
03/15/2022, 6:27 PMKevin Kho
03/15/2022, 6:28 PMAnna Geller
03/15/2022, 6:29 PMSarah Floris
03/15/2022, 6:29 PMAnna Geller
03/15/2022, 6:30 PMSarah Floris
03/15/2022, 6:32 PMAnna Geller
03/15/2022, 6:32 PMdocker login <http://YOUR_REGISTRY_NAME.azurecr.io|YOUR_REGISTRY_NAME.azurecr.io> -u $USER_NAME -p $PASSWORD
Sarah Floris
03/15/2022, 6:32 PMAnna Geller
03/15/2022, 6:33 PMSarah Floris
03/15/2022, 6:37 PMKevin Kho
03/15/2022, 6:41 PMSarah Floris
03/15/2022, 6:43 PMKevin Kho
03/15/2022, 6:45 PMAnna Geller
03/15/2022, 7:05 PMKubernetesRun
but it doesn't mean that you have to use Docker
storage - storage is only to store your Flow definition (so that Prefect can pull it at runtime) while the Docker image is used to package your code dependencies and to define your flow run execution environment (e.g. Python packages that your flow run needs such as pandas, numpy, etc.). You can set a custom Docker image you built yourself on KubernetesRun(image="yourimage")
to define code dependencies for your flow, but for flow storage you can still use e.g. GitHub
storage or Azure
storage - Kevin explained that already but LMK if something is still not clear about Storage.
Handling docker container registry secrets
Once you built your Docker image, you need to reference the image name and tag on your KubernetesRun
run configuration so that Prefect knows which image to use. This image defines your flow run execution environment. But your Kubernetes or Docker agent also need to have proper permissions set up to pull this image at runtime. The way you set those permissions is different for Docker and Kubernetes agents.
#1 To authenticate a DockerAgent
with ACR, you need to run:
docker login <http://YOUR_REGISTRY_NAME.azurecr.io|YOUR_REGISTRY_NAME.azurecr.io> -u $USER_NAME -p $PASSWORD
then you can start your docker agent:
prefect agent docker start --label yourlabel
#2 For KubernetesAgent
, you need to create a Kubernetes Secret which you can create for Azure AKS as follows (note the secret name "aks" here):
kubectl create secret docker-registry aks \
--docker-server=<http://prefectdemos.azurecr.io|prefectdemos.azurecr.io> \
--docker-username=prefectdemos \
--docker-password=$SP_PASSWD
you can use the same username and password as discussed before with the service principal or use this walkthrough https://github.com/anna-geller/packaging-prefect-flows/#azure-container-registry
Then to use this Kubernetes secret on your run config in your flow you can set the secret name we configured above using the argument `image_pull_secrets=["aks"]`:
with Flow(
FLOW_NAME,
storage=STORAGE,
run_config=KubernetesRun(
image="<http://prefectdemos.azurecr.io/community/flows|prefectdemos.azurecr.io/community/flows>",
labels=["aks"],
image_pull_secrets=["aks"], # see README
),
) as flow:
hw = hello_world()