Hey folks, I was wondering if anyone had some docs or advice around the best way setup running agents on an AKS cluster? I am wondering, do I need to package my jobs up to ACR or some other container registers? I’m struggling to identify from the docs how the DockerStorage and the KubernetesRun config is used. I would find a diagram useful here to show the relationship between all of these things. I don’t suppose anyone can ELI5 it for me or had a diagram at hand which details the architecture of it all? Thanks again.

He’s offline. But I’m sure @ciaran would be willing to chime in here. He’s done this.

@Michael Law •

prefect agent kubernetes install

will generate YAML to install an agent on your cluster • If you want to use

DockerStorage

, you'll need to store your images somewhere. You may want to use ACR. Generally, we'd recommend not using

DockerStorage

because it can be slow to build your flows. It's easier to store the flow on

Github

or

S3

and configure your execution image using the

KubernetesRun

• If using

DockerStorage

with a

KubernetesRun

, you do not need to set an image on the

KubernetesRun

-- it will be inferred as the image built by

DockerStorage

• If using another type of storage with a

KubernetesRun

, the default image will just be the

prefect

image. You can set your own base image if you'd like. • You do not need to configure a

KubernetesRun

to run your flows on Kubernetes. If you're not doing any customizations, you can leave the run config empty and it will be given a

UniversalRun

run config. This means that a

DockerStorage

flow will just work on your Kubernetes agent in the image it's stored in and a flow with another storage type will just run on the default image as I described above.

Cool thanks for this @Zanie. I have a couple of things to pitch in with here. You mention an ‘execution image’ what do you mean by that? We are using databricks and databricks-connect too, so I would imagine that will need to be added to my container image too? Am I right in thinking that I would need to build a container which contains my python flows and also my databricks-connect config, so when I trigger a flow it will connect to a specific cluster? This is what will be deployed to kubernetes or am I getting the wrong end of the stick?

Hey @Michael Law we just did this on the

pangeo-forge

project

Awesome man

You might find the

prefect_agent_conf.yaml

useful, I found that the prefect command didn't quite create everything so with some help from @Tyler Wanner we managed to get it working

Haha

(We're not using DockerStorage)

Cool, so if i understood correctly on Kubernetes I thought these ran as jobs? It seems from your config that isnt the case, is that correct?

What do you mean by

these

?

So when we configured a flow, I though the flow would run as a job

Yep it still does

Are you packaging all your flows together and pushing them to a container via that yaml

No, that Yaml is spinning up the Prefect Agent on AKS. Then we're registering flows via the Prefect CLI (Storing them on Azure Blob Storage)

Right think this is where I am getting lost

Yep! If you'd like, I can run you through this on a call? I'll have to get our Azure deployment up but I'm happy to run your through it

That would be unreal man

Cool, I'll get it spun up, make sure I can demo it live without breaking something, then I'll sling a meeting link over!

#hero

No worries! let me know when you're back and I'll ping the link

Thanks man, thats me back

https://meet.google.com/wot-paso-kxy 🦜

Sweet!

I’m not quite sure how the label connects to the databricks cluster. Is it that you have databricks-connect set-up on the agent and it’s configured to a cluster? This means when you run Spark code on an agent, it hits the appropriate Databricks cluster? If that was your thought, I think this looks good! That’s clever. How did you draw this diagram? It looks nice!

That’s exactly it Kevin, the agent jobs are configured with databricks connect and depending on labels will connect to different sized clusters

Glad you got what you were aiming for @Michael Law! The diagram looks great too!