https://prefect.io logo
Powered by
Title
m

Marko Jamedzija

05/31/2021, 1:30 PM
Hi there! I’m trying to run a 
KubernetesRun
so that it runs jobs in a different namespace than the agent is in. I tried setting the namespace in job template yaml, but it doesn’t work. Any suggestions?
I’m getting this error on the agent: 
ERROR:agent:Error while deploying flow
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 420, in deploy_and_update_flow_run
    deployment_info = self.deploy_flow(flow_run)
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 410, in deploy_flow
    namespace=self.namespace, body=job_spec
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api/batch_v1_api.py", line 58, in create_namespaced_job
    (data) = self.create_namespaced_job_with_http_info(namespace, body, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api/batch_v1_api.py", line 149, in create_namespaced_job_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
    _preload_content, _request_timeout)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 166, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 378, in request
    body=body)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 278, in POST
    body=body)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '<some-id>', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Mon, 31 May 2021 13:10:46 GMT', 'Content-Length': '200'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the namespace of the provided object does not match the namespace sent on the request","reason":"BadRequest","code":400}
I see that the agent can be started with 
--namespace <some-namespace>
but the current helm chart I’m using it doesn’t have that option integrated and I don’t want to have my custom override
k

Kevin Kho

05/31/2021, 3:40 PM
Hey @Marko Jamedzija, I'd need to ask the team members with more Kubernetes experience but today is a holiday in the US so I need to get back to you tomorrow
m

Marko Jamedzija

05/31/2021, 3:42 PM
Thanks @Kevin Kho! Looking fw to it 🙂
t

Tyler Wanner

06/02/2021, 4:12 AM
Hi Marko! It appears you're making a request to create a namespaced job that includes 1) a yaml job spec, with a namespace explicitly defined, and 2) a namespace for this job... Since these are not the same namespace, k8s throws this error. the install cli command uses 
--namespace
to set both the agent's namespace and its NAMESPACE env var, but you could alter the output of 
install
to set a different NAMESPACE env var. I wouldn't suggest trying to set namespace in job templates, but would welcome an issue if you have an experience in mind.
m

Marko Jamedzija

06/02/2021, 9:35 AM
Thanks for the explanation Tyler! Do you know if it’s possible to have a different namespace defined for the flow run (
KubernetesRun
)? E.g. I have one agent and I want it to run some flows in one namespace and some in the other.
Or in this case it would be better to use more agents and make a distinction based on run labels?
t

Tyler Wanner

06/02/2021, 1:28 PM
in general, the agent is built to operate within its own namespace (this will impact the resource cleanup also). No, I don't think namespace is something that I would try to manipulate in a run config. Yea, I would suggest running individual agents for each namespace. That's a good use for labels, surely
👍 1
the way i think of it, the agent, its RBAC, and its namespace are setup in Kubernetes as a home for all things related to the use of that agent, whereas run configs define the way agents run flows within that home. That encapsulation is in some ways key to the separation of concerns
m

Marko Jamedzija

06/02/2021, 1:41 PM
Thanks for the clarifications Tyler 🙂
#prefect-serverJoin Slack
Powered by