https://prefect.io logo
Join Slack
Powered by
Hello - Im trying to deploy a flow on Azure K8s, g...
# prefect-kubernetes
p
Hello - Im trying to deploy a flow on Azure K8s, getting the following error while trying to execute the deploy command
Copy code
PS C:\ProjectData\prefect-deploy> prefect deploy -n hello/default
Running deployment build steps...
 > Running build_docker_image step...
Step 1/4 : FROM prefecthq/prefect:2.14.10-python3.11
 ---> d8d0cf17bb52
Step 2/4 : COPY . /opt/prefect/prefect-deploy/
 ---> 534f2a6a6a56
Step 3/4 : WORKDIR /opt/prefect/prefect-deploy/
 ---> Running in 2f7bae63597d
 ---> 0e2b42838ac4
Step 4/4 : LABEL io.prefect.version=2.14.10
 ---> Running in dc11687ab9e1
 ---> 50deb1e1b948
Successfully built 50deb1e1b948
Running deployment push steps...
 > Running push_docker_image step...
The push refers to repository [<http://soaapreg.azurecr.io/hello-flow1|soaapreg.azurecr.io/hello-flow1>]
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Traceback (most recent call last):
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\deployments\steps\core.py", line 154, in run_steps
    step_output = await run_step(step, upstream_outputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\deployments\steps\core.py", line 125, in run_step
    result = await from_async.call_soon_in_new_thread(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\_internal\concurrency\calls.py", line 293, in aresult
    return await asyncio.wrap_future(self.future)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\_internal\concurrency\calls.py", line 318, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect_docker\deployments\steps.py", line 323, in push_docker_image
    raise OSError(event["error"])
OSError: denied: requested access to the resource is denied

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\cli\_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\utilities\asyncutils.py", line 255, in coroutine_wrapper
    return call()
           ^^^^^^
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\_internal\concurrency\calls.py", line 398, in __call__
    return self.result()
           ^^^^^^^^^^^^^
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\_internal\concurrency\calls.py", line 284, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\_internal\concurrency\calls.py", line 168, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\_internal\concurrency\calls.py", line 355, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\cli\deploy.py", line 297, in deploy
    await _run_single_deploy(
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\client\utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\cli\deploy.py", line 607, in _run_single_deploy
    await run_steps(push_steps, step_outputs, print_function=app.console.print)
  File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect\deployments\steps\core.py", line 182, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_docker.deployments.steps.push_docker_image
An exception occurred.
Can some one advice whats the issue here?
n
hi @Parash in the future, please put long tracebacks in threads but this
Copy code
File "C:\Users\ParashHallur\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\prefect_docker\deployments\steps.py", line 323, in push_docker_image
    raise OSError(event["error"])
OSError: denied: requested access to the resource is denied
is often a problem with the repo/image you've supplied for your step can you show the 
push_docker_image
step you defined / are using here?
p
@Nate - Thank you for your response. Following is the "push" section in yaml file.
Copy code
push:
- prefect_docker.deployments.steps.push_docker_image:
    requires: prefect-docker>=0.4.0
    image_name: "{{ build-image.image_name }}"
    tag: "{{ build-image.tag }}"
@Nate - Did you get a chance to check this out?
n
hi - please avoid tagging and keep messages to the thread if possible this is a common error
OSError: denied: requested access to the resource is denied
it means that your 
repo/image
is incorrectly specified as I mentioned in the previous message. I would figure out what the value of 
image_name
and 
tag
and ensure your image repo is included in the 
image_name
p
Thank you again Nate for the response. Here is the build block
Copy code
# build section allows you to manage and build docker images
build:
- prefect_docker.deployments.steps.build_docker_image:
    id: build-image
    requires: prefect-docker>=0.4.0
    image_name: "<http://myimagereg.azurecr.io/hello-flow1|myimagereg.azurecr.io/hello-flow1>"
    tag: latest
    dockerfile: auto
    platform: "linux/amd64"
I see that image is getting built. Im not able to push the image to repo manually also. I tried the "docker login myimagereg.azurecr.io", this went through fine. Even then Im not able to push image manually and also through the deploy.
I was able to resolve by doing docker login with the correct credentials.
Im able to deploy, however the deployment is not running it is failing with the error
Copy code
Failed to submit flow run 'c12472c2-612d-47a8-8d59-1ccccde6b0c6' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 906, in _submit_run_and_capture_errors
    configuration = await self._get_configuration(flow_run)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 986, in _get_configuration
    configuration = await self.job_configuration.from_template_and_values(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 100, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 149, in from_template_and_values
    return cls(**populated_configuration)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pydantic/v1/main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for KubernetesWorkerJobConfiguration
job_manifest
  field required (type=value_error.missing)
This issue is resolved now. I had created Azure Container Instance worker instead of Kubernetes worker
Now Im getting one more error
Copy code
Failed to submit flow run 'a130952c-4667-49e3-9dc8-f055ea6e579d' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 908, in _submit_run_and_capture_errors
    result = await self.run(
             ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 581, in run
    pid = await run_sync_in_worker_thread(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 136, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 859, in _get_infrastructure_pid
    cluster_uid = self._get_cluster_uid(client)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 907, in _get_cluster_uid
    namespace = core_client.read_namespace("kube-system")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 22686, in read_namespace
    return self.read_namespace_with_http_info(name, **kwargs)  # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 22765, in read_namespace_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET
    return self.request("GET", url,
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a2f49be7-5c23-4cbd-8f6e-6ca0b17df112', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'e1308394-e92b-41df-bedf-22a010c83151', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f1660392-31a8-47c7-a82e-ff6f173d865f', 'Date': 'Thu, 20 Jun 2024 18:58:09 GMT', 'Content-Length': '391'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect:prefect-worker\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\": Azure does not have opinion for this user.","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
I resolved this by 1. Deleted this worker 2. created a new service account 3. assigned the cluster binding role 4. mentioned that during the worker deployment
a
Hi, Can you describe your worker deployment configuration ? I am running into the same issue. Did you create a manged identity to assign the role ? Thanks :)
p
Hi Antoine - Used the worker helm chart to deploy and serviceaccount binding with cluster admin role( its not production recommended). Two things added in values.yaml 1. Namespace 2. service account create set to false 3. set the name to newly created serviceaccount in prefect namespace( where worker is going to be part of )
33 Views
Open in Slack
PreviousNext
Powered by