I am working on a POC to use Prefect to help orchestrate an ELT process. I was able to get a flow and tasks to run locally and show up in the local server ui. Then i setup a cloud prefect account because i wanted to test task concurrency and got it to kick of my flow from my local machine. My next step is to see what it takes to productionize the flow in Azure, so i have a few questions. 1. I know there is a azure marketplace item to Create Prefect Agent. Is that the best way to get that part setup? Is Kubernetes the way to go over just a VM with Docker? I tried getting the kubernetes version to install but it failed on the last step with "The received access token is not valid: at least one of the claims 'puid' or 'altsecid' or 'oid' should be present. If you are accessing as application please make sure service principal is properly created in the tenant.". no idea what that means and when i tried to run the install again, it gave me "A custom role with the same name already exists in this directory. Use a different name" 2. Does anyone know of a complete walkthough of setting up an agent in Azure (VM with docker or kubernetes). i did find something, but they dont show anything on setting up the kubernetes service etc. https://infinitelambda.com/post/prefect-workflow-automation-azure-devops-aks/

Hi @Kirk Quinbar, yes feel free to ask in this thread

I’ve just recently setup Prefect Cloud in an Azure environment and highly recommend AKS over VMs/docker/ACI. I utilize

KubernetesRun

to use AKS for both the agent and jobs, with

DockerStorage

and ACR for flows

@Kevin Mullins do you have any documentation on all the steps you did for setting that up? or how did you figure out what all needs to be setup for AKS etc? AKS (and even Docker) are new technologies for me and im not really sure steps to get that setup and then getting it working with Prefect.

@Kirk Quinbar - I really just stitched together a bunch of documentation from the internet. • You’l need a Kubernetes cluster in Azure: Microsoft has some tutorials and documentation here. You’ll want to spend some time familiarizing yourself with Azure and Kubernetes: https://docs.microsoft.com/en-us/azure/aks/tutorial-kubernetes-prepare-app • You’ll need to have a Prefect agent running in Kubernetes: Docs for setting it up are at: https://docs.prefect.io/orchestration/agents/kubernetes.html • Depending on if you want to have Prefect jobs running as a kubernetes job and your local development wants you’ll need to learn about Prefect run configurations. The local run configuration is the starting point that would run your flows in the agent pod. IMHO a better option is the KubernetesRun which will run each job as a Kubernetes job. If you use KubernetesRun, keep in mind how you want your development experience to be. I have my flows always use KubernetesRun and use minikube/kind to run flows locally in Kubernetes. Another option which I’ve seen many do is switch between LocalRun and KubernetesRun based on something like an environment variable for development. Some docs at: https://docs.prefect.io/orchestration/flow_config/run_configs.html