https://prefect.io logo
Channels
Powered by
a

Alan

08/23/2023, 9:30 PM
Hey guys, I am getting this error after helm install prefect/prefect-worker
j

Jamie Zieziula

08/23/2023, 9:49 PM
can you verify that the account & workspace ID are correct?
also, this error might be caused if you don’y have any work pools defined yet? I can’t quite remember
another thing to check is that the key is being loaded into the worker container
a

Alan

08/23/2023, 9:51 PM
ok let me check
Copy code
<https://api.prefect.cloud/api/accounts/49b82480-a4a3-4e5c-aa31-1ea59cbb3866/workspaces/9e97d862-b663-4dc9-aebf-967b6497d329>
Copy code
Environment:
      HOME:                             /home/prefect
      PREFECT_WORKER_PREFETCH_SECONDS:  10
      PREFECT_WORKER_QUERY_SECONDS:     5
      PREFECT_API_ENABLE_HTTP2:         true
      PREFECT_API_URL:                  <https://api.prefect.cloud/api/accounts/49b82480-a4a3-4e5c-aa31-1ea59cbb3866/workspaces/9e97d862-b663-4dc9-aebf-967b6497d329>
      PREFECT_KUBERNETES_CLUSTER_UID:   ed266cf6-xxxx-xxxx-xxxx-87b72959fdb2
      PREFECT_API_KEY:                  pnu_0Oa7rwYFuMdZ9WuCbBGn7xxxxxxxxxxx
      PREFECT_DEBUG_MODE:               false
    Mounts:
      /home/prefect from scratch (rw)
      /tmp from scratch (rw)
image.png
Copy code
Worker 'KubernetesWorker d9666a92-73fb-4be2-9510-f6204d08b35f' started!
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
    return call()
           ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 383, in __call__
    return self.result()
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 283, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 169, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 346, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/cli/worker.py", line 129, in start
    await worker.sync_with_backend()
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 690, in sync_with_backend
    await self._update_local_work_pool_info()
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 647, in _update_local_work_pool_info
    work_pool = await self._client.read_work_pool(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration.py", line 2194, in read_work_pool
    response = await self._client.get(f"/work_pools/{work_pool_name}")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1757, in get
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 280, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 138, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/49b82480-a4a3-4e5c-aa31-1ea59cbb3866/workspaces/9e97d862-b663-4dc9-aebf-967b6497d329/work_pools/>'
Response: {'detail': 'Unauthorized'}
For more information check: <https://httpstatuses.com/403>
An exception occurred.
[ec2-user@ip-172-31-10-44 terraform]$
c

Christopher Boyd

08/23/2023, 10:41 PM
It’s also worth checking that the workspace is correct for the api key (not just the account), and that the account that generated the api key has appropriate RBAC permissions / role assigned (if you’re using something other than a standard configuration)
a

Alan

08/23/2023, 10:42 PM
how can i get api key for the workspace?
c

Christopher Boyd

08/23/2023, 10:42 PM
api keys are generated on a per workspace basis
a workspace might be “dev”, “stage”, and “prod”, or “teamA”, and “teamB” - they don’t have reach across workspaces
a

Alan

08/23/2023, 10:43 PM
this is my workspace id
9e97d862-b663-4dc9-aebf-967b6497d329
how can i make an api key targeting it
c

Christopher Boyd

08/23/2023, 10:44 PM
you can check this article to verify your api key and URL are indeed accurate - https://discourse.prefect.io/t/how-can-i-tell-if-there-is-an-issue-with-my-api-key/2185
a

Alan

08/23/2023, 10:44 PM
I tried that
c

Christopher Boyd

08/23/2023, 10:44 PM
from there, it’s likely either a missing work pool as Jamie mentioned, or RBAC / role assignment permissions for the user that generated that key
a

Alan

08/23/2023, 10:44 PM
I get unauthorized
c

Christopher Boyd

08/23/2023, 10:44 PM
that’s the issue then - you need to generate a new API key for your workspace
image.png
go to your profile and create a new api key for that workspace
a

Alan

08/23/2023, 10:45 PM
no where in the ui does it let you specify a workspace for the new api key
c

Christopher Boyd

08/23/2023, 10:46 PM
can you try generating a new api key, and testing with that?
a

Alan

08/23/2023, 10:46 PM
I did that multiple times
image.png
with the same result
c

Christopher Boyd

08/23/2023, 10:47 PM
the account id and workspace id - where are you retrieving those values from?
a

Alan

08/23/2023, 10:47 PM
image.png
c

Christopher Boyd

08/23/2023, 10:48 PM
your account id is incorrect
it shows you’re using one beginning with 
49
, but the one in your screenshot is showing 
7d
a

Alan

08/23/2023, 10:49 PM
Copy code
Environment:
      HOME:                             /home/prefect
      PREFECT_WORKER_PREFETCH_SECONDS:  10
      PREFECT_WORKER_QUERY_SECONDS:     5
      PREFECT_API_ENABLE_HTTP2:         true
      PREFECT_API_URL:                  <https://api.prefect.cloud/api/accounts/49b82480-a4a3-4e5c-aa31-1ea59cbb3866/workspaces/9e97d862-b663-4dc9-aebf-967b6497d329>
      PREFECT_KUBERNETES_CLUSTER_UID:   ed266cf6-xxxx-xxxx-xxxx-87b72959fdb2
      PREFECT_API_KEY:                  pnu_0Oa7rwYFuMdZ9WuCbBGn7xxxxxxxxxxx
      PREFECT_DEBUG_MODE:               false
    Mounts:
      /home/prefect from scratch (rw)
      /tmp from scratch (rw)
7d?
where are you seeing 7d?
c

Christopher Boyd

08/23/2023, 10:50 PM
yes - the image you just shared shows an account id of 
7d57882b-0dd8-4a3a-844a-991b6a469881
it’s in your url
a

Alan

08/23/2023, 10:51 PM
I see
but I got the account id from here
image.png
c

Christopher Boyd

08/23/2023, 10:52 PM
that’s for your individual user account, not for the organizational
you can be part of several organizations each with an accountID
I think this has come up a few times however, and there is an enhancement request to propose changing the nomenclature of this identifier
each user receives a user account ID, which is distinct and separate from your prefect account (which might be individual, or might be a company wide organization)
a

Alan

08/23/2023, 10:54 PM
ok
now I get this
Copy code
File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 280, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 138, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '405 Method Not Allowed' for url '<https://api.prefect.cloud/api/accounts/7d57882b-0dd8-4a3a-844a-991b6a469881/workspaces/9e97d862-b663-4dc9-aebf-967b6497d329/work_pools/>'
Response: {'detail': 'Method Not Allowed'}
For more information check: <https://httpstatuses.com/405>
An exception occurred.
[ec2-user@ip-172-31-10-44 terraform]$ ^C
[ec2-user@ip-172-31-10-44 terraform]$
method not allowed
c

Christopher Boyd

08/23/2023, 10:55 PM
I’m not really sure of the rest of what’s happening here from just that
or the worker that you’re using
a

Alan

08/23/2023, 10:55 PM
method not allowed is from your api for sure
Copy code
<https://api.prefect.cloud/api/accounts/7d57882b-0dd8-4a3a-844a-991b6a469881/workspaces/9e97d862-b663-4dc9-aebf-967b6497d329/work_pools/>
fixed it
j

Jamie Zieziula

08/23/2023, 11:15 PM
checking in - did everything get resolved?
a

Alan

08/24/2023, 8:55 PM
yes it did
have new question
Copy code
Worker 'KubernetesWorker 366cb317-e798-4d29-bb52-dc9e1ccddc5b' submitting flow run 'a07be205-1805-493c-9221-af2e1c992666'
01:47:43 PM
prefect.flow_runs.worker
Creating Kubernetes job...
01:47:43 PM
prefect.flow_runs.worker
Failed to submit flow run 'a07be205-1805-493c-9221-af2e1c992666' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 628, in _create_job
    job = batch_client.create_namespaced_job(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/batch_v1_api.py", line 210, in create_namespaced_job
    return self.create_namespaced_job_with_http_info(namespace, body, **kwargs)  # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/batch_v1_api.py", line 309, in create_namespaced_job_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 391, in request
    return <http://self.rest_client.POST|self.rest_client.POST>(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 276, in POST
    return self.request("POST", url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 235, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '1ac0431a-c6d1-43de-a14e-e70676323d23', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '27e0cf44-f342-4851-b226-266eea1ccee4', 'X-Kubernetes-Pf-Prioritylevel-Uid': '7e3a6a21-766e-4d8c-ad7e-541a352ac89c', 'Date': 'Thu, 24 Aug 2023 20:47:43 GMT', 'Content-Length': '290'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:anonymous\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
seeing that its using system:anonymous as the service account
why isn't it using the prefect-worker service account?
@Christopher Boyd
@Jamie Zieziula
j

Jamie Zieziula

08/24/2023, 9:19 PM
in your workpool settings, did you define the service account to us?
Screenshot 2023-08-24 at 5.19.53 PM.png
a

Alan

08/25/2023, 4:54 PM
Yes I did
image.png
image.png
I Installed the worker through helm
and that service-account seems to have ability to create batch jobs
j

Jamie Zieziula

08/25/2023, 4:56 PM
what is the kubeconfig you have provided to the worker?
i would remove that and allow the worker to use the native connection information
a

Alan

08/25/2023, 4:57 PM
how do I remove that?
just by selecting none?
image.png
j

Jamie Zieziula

08/25/2023, 5:15 PM
yes
a

Alan

08/25/2023, 6:09 PM
tried it
not working
image.png
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/deployments/steps/core.py", line 124, in run_steps
    step_output = await run_step(step, upstream_outputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/deployments/steps/core.py", line 95, in run_step
    result = await from_async.call_soon_in_new_thread(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 292, in aresult
    return await asyncio.wrap_future(self.future)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 316, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/deployments/steps/pull.py", line 28, in set_working_directory
    os.chdir(directory)
FileNotFoundError: [Errno 2] No such file or directory: '/home/pythonking/ML-DEVOPS/pipelines/prefect-pipelines/example-test'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 395, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/deployments/deployments.py", line 207, in load_flow_from_flow_run
    output = await run_steps(deployment.pull_steps)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/deployments/steps/core.py", line 152, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.set_working_directory
Copy code
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.set_working_directory
Copy code
FileNotFoundError: [Errno 2] No such file or directory: '/home/pythonking/ML-DEVOPS/pipelines/prefect-pipelines/example-test'
Copy code
###
### A complete description of a Prefect Deployment for flow 'branching-flow'
###
name: alan-test
description: null
version: bbbd9c765796684fa2941954f530638e
# The work queue that will handle this deployment's runs
work_queue_name: ili-pool
work_pool_name: kubeflow
tags: []
parameters: {}
schedule: null
is_schedule_active: null
infra_overrides: {}
infrastructure:
  type: process
  env: {}
  labels: {}
  name: null
  command: null
  stream_output: true
  working_dir: "./"
  block_type_slug: process
  _block_type_slug: process

###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: branching-flow
manifest_path: null
storage: null
path: /home/pythonking/ML-DEVOPS/pipelines/prefect-pipelines/example-test
entrypoint: fork_and_join.py:branching_flow
parameter_openapi_schema:
  title: Parameters
  type: object
  properties:
    slice_count:
      title: slice_count
      position: 0
      type: integer
    slice_by:
      title: slice_by
      default: 23
      position: 1
      type: integer
  required:
  - slice_count
  definitions: null
timestamp: '2023-08-23T23:50:46.681710+00:00'
triggers: []
this is my deployment yaml
j

Jamie Zieziula

08/25/2023, 8:03 PM
Ah, I think you will need to upgrade to the new prefect.yaml for deployments in order for it to work with 
workers
a

Alan

09/12/2023, 5:24 PM
Hello Jamie
I am just wondering! What would I be losing if i installed the prefect dashboard/server locally vs paying for prefect cloud?
@Jamie Zieziula
c

Christopher Boyd

09/12/2023, 6:53 PM
Hi Alan, you can reach out to our product advocates who would be happy to assist you with these questions - @Bianca Hoch and @Sahil Rangwala
✅ 2
10 Views
Powered by