Hi all! We have an EKS cluster in which we would like to run our flows. We managed to build an architecture purely based on fargate but for our purposes we moved to an hybrid cluster, in which there are both managed nodegroups and a fargate profile. I would like to have a small node (identified by -ns prefect-agent) for my prefect agent which runs jobs on the fargate profile (identified by -ns serverless). I thought it was enough to specify the different namespace in the job_template of the KubernetesRun, by the way the flow run fails once started with a 400 error and the message: "the namespace of the provided object does not match the namespace sent on the request". I cannot find anything useful about running flows in a different namespace from the agent. Does anyone have managed something similar? Thanks in advance

Wow, interesting use case and nice architecture! Have you tried spinning up one agent for each? This would potentially make things much easier since a flow with run config

KubernetesRun(labels=["serverless"])

could be picked up by the agent living in the Fargate-based EKS data plane and "serverless" namespace while

KubernetesRun(labels=["prefect-agent"])

could be picked up by the agent running in "prefect-agent" namespace with managed nodegroups. Of course, this required 2 agent pods running at all times, but would make it much easier to write flows.

Hi Anna, thank you for your answer. No, I didn't tried to spin 2 agents. My goal was to use a single agent in -ns prefect-agent to manage all my flows, e.g. data-eng flows on -ns serverless because they only run for a couple of minutes every 6 hours; ML training flows on -ns gpu-node because they need a gpu based node which is not available in fargate. The architecture you proposed implies a different agent for each namespace, correct?

exactly, I think it can make things much easier