Hey, I am receiving this error after running. Also the agent serverless kubernetes cluster is not sh...
s

Sagun Garg

over 4 years ago
Hey, I am receiving this error after running. Also the agent serverless kubernetes cluster is not showing up in the Prefect UI dashboard @Scott Zelenka and @Pedro Martins have faced similar issues before linking their previous conversations here https://prefect-community.slack.com/archives/CL09KU1K7/p1596036675380000 and https://prefect-community.slack.com/archives/C014Z8DPDSR/p1607966613290400
$ prefect agent kubernetes install -t <MY_TOKEN> --rbac | kubectl apply -f -
Warning Unhealthy kubelet  Liveness probe failed :8080/api/health Logs of the failing pod made to restart by K8s
Warning LoggingDisabled 48m          fargate-scheduler Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
 
Normal  Scheduled    48m          fargate-scheduler Successfully assigned default/prefect-agent-7bcdb4f975-pftnw to fargate-ip-10-0-231-193.ap-southeast-1.compute.internal
 
Normal  Created     43m (x4 over 47m)   kubelet      Created container agent
 
Normal  Started     43m (x4 over 47m)   kubelet      Started container agent
 
Normal  Killing     42m (x2 over 45m)   kubelet      Container agent failed liveness probe, will be restarted
 
Normal  Pulling     41m (x5 over 48m)   kubelet      Pulling image "prefecthq/prefect:0.14.0-python3.6"
 
Normal  Pulled      41m (x5 over 47m)   kubelet      Successfully pulled image "prefecthq/prefect:0.14.0-python3.6"
 
Warning Unhealthy    12m (x13 over 46m)  kubelet      Liveness probe failed: Get <http://10.0.231.193:8080/api/health>: dial tcp 10.0.231.193:8080: connect: connection refused
 
Warning BackOff     3m3s (x113 over 43m) kubelet      Back-off restarting failed container
Hi! I’ve been encountering below error with self-hosted prefect server. Have verified worker was dep...
c

Christine Chen

8 months ago
Hi! I’ve been encountering below error with self-hosted prefect server. Have verified worker was deployed to the same namespace. Any ideas? Prefect version: 3.0.4 Server/worker chart version: 2024.10.1175730
Failed to submit flow run '46744906-60cd-410c-a04c-12d7a80917cf' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 985, in _submit_run_and_capture_errors
    await self._give_worker_labels_to_flow_run(flow_run.id)
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 1232, in _give_worker_labels_to_flow_run
    await self._client.update_flow_run_labels(flow_run_id, labels)
  File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration/__init__.py", line 2950, in update_flow_run_labels
    response = await self._client.patch(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1933, in patch
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1540, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 355, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 163, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<http://prefect-server>.{namespace}.svc.cluster.local:4200/api/flow_runs/46744906-60cd-410c-a04c-12d7a80917cf/labels'
Response: {'detail': 'Not Found'}
1