Hi folks, I'm running Prefect Server in my Kubernetes cluster with a Kubernetes agent. I'm also running a permanent Dask cluster in Kubernetes which I use for running parallelised flows. But each job passes through the Kubernetes Agent on its way to the Dask cluster necessitating the creation of a Kubernetes Job object before the Dask cluster work is scheduled. How should I get around this? Can I run a local agent on the GraphQL server? Thanks all!

Hi @Aiden Price - you can run any number (or type) of agent that you'd like! Running a local agent will create a subprocess to submit the run to the dask cluster rather than using a k8s job, so that should do what you're hoping. You won't want to run your local agent on the graphql server but you can spin up the agent as a separate node on your k8s cluster

So I create a new pod with

prefect agent local

instead of

prefect agent kubernetes

? How do the two pods communicate then? Isn't it assumed that the local agent's address is

localhost

?

Nope, you can change which address the agent queries for runs

You can use the env variables. We are injecting then at run time like PREFECT___SERVER___HOST=http://prefect PREFECT__BACKEND=server All the parameters of config.toml can be set as end variable

Okay, thank you. I'll try and implement it today and tomorrow and let you know if I hit any issues.