Alan
08/23/2023, 9:30 PMJamie Zieziula
08/23/2023, 9:49 PMJamie Zieziula
08/23/2023, 9:49 PMJamie Zieziula
08/23/2023, 9:50 PMAlan
08/23/2023, 9:51 PMAlan
08/23/2023, 9:52 PM<https://api.prefect.cloud/api/accounts/49b82480-a4a3-4e5c-aa31-1ea59cbb3866/workspaces/9e97d862-b663-4dc9-aebf-967b6497d329>
Alan
08/23/2023, 9:54 PMEnvironment:
HOME: /home/prefect
PREFECT_WORKER_PREFETCH_SECONDS: 10
PREFECT_WORKER_QUERY_SECONDS: 5
PREFECT_API_ENABLE_HTTP2: true
PREFECT_API_URL: <https://api.prefect.cloud/api/accounts/49b82480-a4a3-4e5c-aa31-1ea59cbb3866/workspaces/9e97d862-b663-4dc9-aebf-967b6497d329>
PREFECT_KUBERNETES_CLUSTER_UID: ed266cf6-xxxx-xxxx-xxxx-87b72959fdb2
PREFECT_API_KEY: pnu_0Oa7rwYFuMdZ9WuCbBGn7xxxxxxxxxxx
PREFECT_DEBUG_MODE: false
Mounts:
/home/prefect from scratch (rw)
/tmp from scratch (rw)
Alan
08/23/2023, 9:55 PMAlan
08/23/2023, 10:41 PMWorker 'KubernetesWorker d9666a92-73fb-4be2-9510-f6204d08b35f' started!
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
return call()
^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 383, in __call__
return self.result()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 283, in result
return self.future.result(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 169, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 346, in _run_async
result = await coro
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/cli/worker.py", line 129, in start
await worker.sync_with_backend()
File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 690, in sync_with_backend
await self._update_local_work_pool_info()
File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 647, in _update_local_work_pool_info
work_pool = await self._client.read_work_pool(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration.py", line 2194, in read_work_pool
response = await self._client.get(f"/work_pools/{work_pool_name}")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1757, in get
return await self.request(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 280, in send
response.raise_for_status()
File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 138, in raise_for_status
raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/49b82480-a4a3-4e5c-aa31-1ea59cbb3866/workspaces/9e97d862-b663-4dc9-aebf-967b6497d329/work_pools/>'
Response: {'detail': 'Unauthorized'}
For more information check: <https://httpstatuses.com/403>
An exception occurred.
[ec2-user@ip-172-31-10-44 terraform]$
Christopher Boyd
08/23/2023, 10:41 PMChristopher Boyd
08/23/2023, 10:42 PMAlan
08/23/2023, 10:42 PMChristopher Boyd
08/23/2023, 10:42 PMChristopher Boyd
08/23/2023, 10:42 PMAlan
08/23/2023, 10:43 PMAlan
08/23/2023, 10:43 PMAlan
08/23/2023, 10:43 PMChristopher Boyd
08/23/2023, 10:44 PMAlan
08/23/2023, 10:44 PMChristopher Boyd
08/23/2023, 10:44 PMAlan
08/23/2023, 10:44 PMChristopher Boyd
08/23/2023, 10:44 PMChristopher Boyd
08/23/2023, 10:45 PMChristopher Boyd
08/23/2023, 10:45 PMAlan
08/23/2023, 10:45 PMChristopher Boyd
08/23/2023, 10:46 PMAlan
08/23/2023, 10:46 PMAlan
08/23/2023, 10:46 PMAlan
08/23/2023, 10:46 PMChristopher Boyd
08/23/2023, 10:47 PMAlan
08/23/2023, 10:47 PMChristopher Boyd
08/23/2023, 10:48 PMChristopher Boyd
08/23/2023, 10:48 PM49
, but the one in your screenshot is showing 7d
Alan
08/23/2023, 10:49 PMEnvironment:
HOME: /home/prefect
PREFECT_WORKER_PREFETCH_SECONDS: 10
PREFECT_WORKER_QUERY_SECONDS: 5
PREFECT_API_ENABLE_HTTP2: true
PREFECT_API_URL: <https://api.prefect.cloud/api/accounts/49b82480-a4a3-4e5c-aa31-1ea59cbb3866/workspaces/9e97d862-b663-4dc9-aebf-967b6497d329>
PREFECT_KUBERNETES_CLUSTER_UID: ed266cf6-xxxx-xxxx-xxxx-87b72959fdb2
PREFECT_API_KEY: pnu_0Oa7rwYFuMdZ9WuCbBGn7xxxxxxxxxxx
PREFECT_DEBUG_MODE: false
Mounts:
/home/prefect from scratch (rw)
/tmp from scratch (rw)
Alan
08/23/2023, 10:50 PMAlan
08/23/2023, 10:50 PMChristopher Boyd
08/23/2023, 10:50 PM7d57882b-0dd8-4a3a-844a-991b6a469881
Christopher Boyd
08/23/2023, 10:50 PMAlan
08/23/2023, 10:51 PMAlan
08/23/2023, 10:51 PMAlan
08/23/2023, 10:51 PMChristopher Boyd
08/23/2023, 10:52 PMChristopher Boyd
08/23/2023, 10:53 PMChristopher Boyd
08/23/2023, 10:53 PMChristopher Boyd
08/23/2023, 10:54 PMAlan
08/23/2023, 10:54 PMAlan
08/23/2023, 10:54 PMAlan
08/23/2023, 10:54 PMFile "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 280, in send
response.raise_for_status()
File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 138, in raise_for_status
raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '405 Method Not Allowed' for url '<https://api.prefect.cloud/api/accounts/7d57882b-0dd8-4a3a-844a-991b6a469881/workspaces/9e97d862-b663-4dc9-aebf-967b6497d329/work_pools/>'
Response: {'detail': 'Method Not Allowed'}
For more information check: <https://httpstatuses.com/405>
An exception occurred.
[ec2-user@ip-172-31-10-44 terraform]$ ^C
[ec2-user@ip-172-31-10-44 terraform]$
Alan
08/23/2023, 10:55 PMChristopher Boyd
08/23/2023, 10:55 PMChristopher Boyd
08/23/2023, 10:55 PMAlan
08/23/2023, 10:55 PMAlan
08/23/2023, 10:56 PM<https://api.prefect.cloud/api/accounts/7d57882b-0dd8-4a3a-844a-991b6a469881/workspaces/9e97d862-b663-4dc9-aebf-967b6497d329/work_pools/>
Alan
08/23/2023, 11:08 PMJamie Zieziula
08/23/2023, 11:15 PMAlan
08/24/2023, 8:55 PMAlan
08/24/2023, 8:55 PMAlan
08/24/2023, 8:56 PMWorker 'KubernetesWorker 366cb317-e798-4d29-bb52-dc9e1ccddc5b' submitting flow run 'a07be205-1805-493c-9221-af2e1c992666'
01:47:43 PM
prefect.flow_runs.worker
Creating Kubernetes job...
01:47:43 PM
prefect.flow_runs.worker
Failed to submit flow run 'a07be205-1805-493c-9221-af2e1c992666' to infrastructure.
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 628, in _create_job
job = batch_client.create_namespaced_job(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/batch_v1_api.py", line 210, in create_namespaced_job
return self.create_namespaced_job_with_http_info(namespace, body, **kwargs) # noqa: E501
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/batch_v1_api.py", line 309, in create_namespaced_job_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 391, in request
return <http://self.rest_client.POST|self.rest_client.POST>(url,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 276, in POST
return self.request("POST", url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 235, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '1ac0431a-c6d1-43de-a14e-e70676323d23', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '27e0cf44-f342-4851-b226-266eea1ccee4', 'X-Kubernetes-Pf-Prioritylevel-Uid': '7e3a6a21-766e-4d8c-ad7e-541a352ac89c', 'Date': 'Thu, 24 Aug 2023 20:47:43 GMT', 'Content-Length': '290'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:anonymous\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
Alan
08/24/2023, 8:56 PMAlan
08/24/2023, 8:56 PMAlan
08/24/2023, 8:58 PMAlan
08/24/2023, 8:58 PMJamie Zieziula
08/24/2023, 9:19 PMJamie Zieziula
08/24/2023, 9:19 PMAlan
08/25/2023, 4:54 PMAlan
08/25/2023, 4:54 PMAlan
08/25/2023, 4:55 PMAlan
08/25/2023, 4:56 PMAlan
08/25/2023, 4:56 PMJamie Zieziula
08/25/2023, 4:56 PMJamie Zieziula
08/25/2023, 4:57 PMAlan
08/25/2023, 4:57 PMAlan
08/25/2023, 4:57 PMAlan
08/25/2023, 4:57 PMJamie Zieziula
08/25/2023, 5:15 PMAlan
08/25/2023, 6:09 PMAlan
08/25/2023, 6:09 PMAlan
08/25/2023, 6:10 PMAlan
08/25/2023, 6:11 PMFlow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/prefect/deployments/steps/core.py", line 124, in run_steps
step_output = await run_step(step, upstream_outputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/deployments/steps/core.py", line 95, in run_step
result = await from_async.call_soon_in_new_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 292, in aresult
return await asyncio.wrap_future(self.future)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 316, in _run_sync
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/deployments/steps/pull.py", line 28, in set_working_directory
os.chdir(directory)
FileNotFoundError: [Errno 2] No such file or directory: '/home/pythonking/ML-DEVOPS/pipelines/prefect-pipelines/example-test'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 395, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
return await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/deployments/deployments.py", line 207, in load_flow_from_flow_run
output = await run_steps(deployment.pull_steps)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/deployments/steps/core.py", line 152, in run_steps
raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.set_working_directory
Alan
08/25/2023, 6:15 PMprefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.set_working_directory
Alan
08/25/2023, 6:21 PMFileNotFoundError: [Errno 2] No such file or directory: '/home/pythonking/ML-DEVOPS/pipelines/prefect-pipelines/example-test'
Alan
08/25/2023, 6:21 PM###
### A complete description of a Prefect Deployment for flow 'branching-flow'
###
name: alan-test
description: null
version: bbbd9c765796684fa2941954f530638e
# The work queue that will handle this deployment's runs
work_queue_name: ili-pool
work_pool_name: kubeflow
tags: []
parameters: {}
schedule: null
is_schedule_active: null
infra_overrides: {}
infrastructure:
type: process
env: {}
labels: {}
name: null
command: null
stream_output: true
working_dir: "./"
block_type_slug: process
_block_type_slug: process
###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: branching-flow
manifest_path: null
storage: null
path: /home/pythonking/ML-DEVOPS/pipelines/prefect-pipelines/example-test
entrypoint: fork_and_join.py:branching_flow
parameter_openapi_schema:
title: Parameters
type: object
properties:
slice_count:
title: slice_count
position: 0
type: integer
slice_by:
title: slice_by
default: 23
position: 1
type: integer
required:
- slice_count
definitions: null
timestamp: '2023-08-23T23:50:46.681710+00:00'
triggers: []
Alan
08/25/2023, 6:22 PMJamie Zieziula
08/25/2023, 8:03 PMworkers
Jamie Zieziula
08/25/2023, 8:03 PMAlan
09/12/2023, 5:24 PMAlan
09/12/2023, 5:25 PMAlan
09/12/2023, 6:34 PMChristopher Boyd
09/12/2023, 6:53 PM