Thread
#prefect-server
    m

    Marko Jamedzija

    1 year ago
    Hi there! I’m trying to run a
    KubernetesRun
    so that it runs jobs in a different namespace than the agent is in. I tried setting the namespace in job template yaml, but it doesn’t work. Any suggestions?
    I’m getting this error on the agent:
    ERROR:agent:Error while deploying flow
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 420, in deploy_and_update_flow_run
        deployment_info = self.deploy_flow(flow_run)
      File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 410, in deploy_flow
        namespace=self.namespace, body=job_spec
      File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api/batch_v1_api.py", line 58, in create_namespaced_job
        (data) = self.create_namespaced_job_with_http_info(namespace, body, **kwargs)  # noqa: E501
      File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api/batch_v1_api.py", line 149, in create_namespaced_job_with_http_info
        collection_formats=collection_formats)
      File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
        _preload_content, _request_timeout)
      File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 166, in __call_api
        _request_timeout=_request_timeout)
      File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 378, in request
        body=body)
      File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 278, in POST
        body=body)
      File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in request
        raise ApiException(http_resp=r)
    kubernetes.client.rest.ApiException: (400)
    Reason: Bad Request
    HTTP response headers: HTTPHeaderDict({'Audit-Id': '<some-id>', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Mon, 31 May 2021 13:10:46 GMT', 'Content-Length': '200'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the namespace of the provided object does not match the namespace sent on the request","reason":"BadRequest","code":400}
    I see that the agent can be started with
    --namespace <some-namespace>
    but the current helm chart I’m using it doesn’t have that option integrated and I don’t want to have my custom override
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Marko Jamedzija, I'd need to ask the team members with more Kubernetes experience but today is a holiday in the US so I need to get back to you tomorrow
    m

    Marko Jamedzija

    1 year ago
    Thanks @Kevin Kho! Looking fw to it 🙂
    Tyler Wanner

    Tyler Wanner

    1 year ago
    Hi Marko! It appears you're making a request to create a namespaced job that includes 1) a yaml job spec, with a namespace explicitly defined, and 2) a namespace for this job... Since these are not the same namespace, k8s throws this error. the install cli command uses
    --namespace
    to set both the agent's namespace and its NAMESPACE env var, but you could alter the output of
    install
    to set a different NAMESPACE env var. I wouldn't suggest trying to set namespace in job templates, but would welcome an issue if you have an experience in mind.
    m

    Marko Jamedzija

    1 year ago
    Thanks for the explanation Tyler! Do you know if it’s possible to have a different namespace defined for the flow run (
    KubernetesRun
    )? E.g. I have one agent and I want it to run some flows in one namespace and some in the other.
    Or in this case it would be better to use more agents and make a distinction based on run labels?
    Tyler Wanner

    Tyler Wanner

    1 year ago
    in general, the agent is built to operate within its own namespace (this will impact the resource cleanup also). No, I don't think namespace is something that I would try to manipulate in a run config. Yea, I would suggest running individual agents for each namespace. That's a good use for labels, surely
    the way i think of it, the agent, its RBAC, and its namespace are setup in Kubernetes as a home for all things related to the use of that agent, whereas run configs define the way agents run flows within that home. That encapsulation is in some ways key to the separation of concerns
    m

    Marko Jamedzija

    1 year ago
    Thanks for the clarifications Tyler 🙂