Hi folks, I'm running Prefect Server in my Kuberne...
# prefect-server
Hi folks, I'm running Prefect Server in my Kubernetes cluster with a Kubernetes agent. I'm also running a permanent Dask cluster in Kubernetes which I use for running parallelised flows. But each job passes through the Kubernetes Agent on its way to the Dask cluster necessitating the creation of a Kubernetes Job object before the Dask cluster work is scheduled. How should I get around this? Can I run a local agent on the GraphQL server? Thanks all!
Hi @Aiden Price - you can run any number (or type) of agent that you'd like! Running a local agent will create a subprocess to submit the run to the dask cluster rather than using a k8s job, so that should do what you're hoping. You won't want to run your local agent on the graphql server but you can spin up the agent as a separate node on your k8s cluster
So I create a new pod with
prefect agent local
instead of
prefect agent kubernetes
? How do the two pods communicate then? Isn't it assumed that the local agent's address is
Also thank you for getting back to me so quickly.
Nope, you can change which address the agent queries for runs
One sec and I can find that config
Ok you should be able to set this in the
in your agent pod :
Copy code
api = '<<your graphql endpoint, defaults to localhost:4200/graphql>>
that heading might be server instead of cloud though i'm not sure 🤔
You can use the env variables. We are injecting then at run time like PREFECT___SERVER___HOST=http://prefect PREFECT__BACKEND=server All the parameters of config.toml can be set as end variable
Okay, thank you. I'll try and implement it today and tomorrow and let you know if I hit any issues.