Kiley Roberson
11/30/2023, 10:22 PMbase-prefect
.
pull:
- prefect.deployments.steps.git_clone: &clone_repo
repository: my_repo
branch: test-deploy-branch
access_token: token_in_secrets
definitions:
actions:
pull_and_setup: &pull_and_setup
- prefect.deployments.steps.git_clone:
<<: *clone_repo
- base-prefect.utilities.secrets.retrieve_secrets.service_secrets: <-- this is throwing errors
id: retrieve_secrets
Here is the flow deployment
- name: my flow
version:
tags: []
description:
entrypoint: flows/my_flow.py:my_flow
parameters
work_pool:
name: kubernetes-work-pool
work_queue_name:
job_variables:
image: us-docker.pkg.dev/project/prefect/base_image:latest
schedule:
pull: *pull_and_setup
With the older image I have this works fine - but when I build and push up a new image I get that error. So Im wondering if maybe something changed with the prefecthq/prefect:2-python3.11
base image?
on the old image if I print out the current working dir I get /opt/prefect/base-prefect
but with the new image I get /opt/prefect/base-prefect-test-deploy-branch
- so Im not sure why it is appending my branch name to this and I am wondering if this is causing the problems.
Thank you for any help I can get on this!Don
12/05/2023, 12:02 AMAndrew
12/05/2023, 4:51 PMhelm install
will work if its within the same namespace, so I tried running it in a new namespace, but its just showing Error from server (BadRequest): container "prefect-worker" in pod "prefect-worker-8678675458-kznph" is waiting to start: CreateContainerConfigError
in the worker pod logs.
Any advice on the general setup for multiple worker pools inside k8s?Gregory Hunt
12/11/2023, 8:43 PMMax Eggers
12/12/2023, 11:47 PMKrystal
12/14/2023, 2:27 PMsqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
in the server. Any idea how to resolve this please?Gregory Hunt
12/14/2023, 9:00 PMTom Matthews
12/15/2023, 9:48 AMMax Eggers
12/15/2023, 3:15 PMOfir Or
12/18/2023, 7:46 AMnicholasnet
12/18/2023, 9:01 PMTraceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 260, in coroutine_wrapper
return call()
^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 245, in __call__
return self.result()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 173, in result
return self.future.result(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
result = await coro
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/cli/deployment.py", line 1143, in build
deployment = await Deployment.build_from_flow(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/deployments.py", line 783, in build_from_flow
await deployment.load()
File "/usr/local/lib/python3.11/site-packages/prefect/deployments.py", line 559, in load
deployment = await client.read_deployment_by_name(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration.py", line 1513, in read_deployment_by_name
return schemas.responses.DeploymentResponse.parse_obj(response.json())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 8 validation errors for DeploymentResponse
schedule -> interval
field required (type=value_error.missing)
schedule -> timezone
argument of type 'function' is not iterable (type=type_error)
schedule -> cron
extra fields not permitted (type=value_error.extra)
schedule -> day_or
extra fields not permitted (type=value_error.extra)
schedule -> timezone
argument of type 'function' is not iterable (type=type_error)
schedule -> rrule
field required (type=value_error.missing)
schedule -> cron
extra fields not permitted (type=value_error.extra)
schedule -> day_or
extra fields not permitted (type=value_error.extra)
An exception occurred.
Is anybody else having this issue?Scott Melhop
12/20/2023, 9:28 AMKrystal
01/04/2024, 4:51 PMjob_config
in the workerpool. The way I am deploying a workflow is through Python by calling .deploy()
. However, by passing the json which contains the job config doesn't really change the manifest. Can someone give me a working example of this please?Xing Zeng
01/11/2024, 10:38 PMDave
01/15/2024, 11:08 AMFailed to establish a new connection: [Errno 99] Cannot assign requested address
issue with prefect workers?
Currently we have to periodically restart our workers in order to avoid the work pool getting stuck on that error. Unfortunately that causes other issues as Prefect is apparently not able to gracefully handle worker restarts while it has flows in "Pending"/"Cancelling"/"Running" states. These flows will stay in those states forever as their worker doesn't exist anymore and they still count towards the work pool concurrency limit so everything just runs to a screeching halt. I have to run a manual script to force delete/reschedule these stuck flows and release the concurrency limit slots.
How are people using Prefect work pools without the need of constant human supervision and manual scripts?
Related issues:
https://github.com/PrefectHQ/prefect-kubernetes/issues/102
https://discourse.prefect.io/t/flow-stuck-in-pending-mode/3406Tom Matthews
01/18/2024, 12:33 PMJob 'ultraviolet-lyrebird-vhjs7': Job reached backoff limit.
Andy Irvine
01/29/2024, 9:37 PMNick DeCraene
02/09/2024, 7:41 PMYing Ting Loo
02/14/2024, 6:53 AMMikaël Ferreira de Almeida
02/16/2024, 12:51 PMautoscaling:
enabled: true
minReplicas: 1
maxReplicas: 5
targetCPU: 90
targetMemory: 90
resources:
requests:
cpu: 500m
memory: 256Mi
limits:
cpu: 1000m
memory: 512Mi
And when I look at HPA i see this ⬇️ , high CPU usage
> kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
prefect-server Deployment/prefect-server 131%/90%, 2%/90% 1 5 5 20d
But my pods are not using CPU:
> kubectl top pods
NAME CPU(cores) MEMORY(bytes)
prefect-server-7f59dcf988-5zdx6 9m 332Mi
prefect-server-7f59dcf988-72d7q 7m 339Mi
prefect-server-7f59dcf988-pxnvr 12m 335Mi
prefect-server-7f59dcf988-szt9r 10m 343Mi
prefect-server-7f59dcf988-zm6vd 15m 327Mi
Does anyone have the same problem?Sean Davis
02/19/2024, 4:15 PMYing Ting Loo
02/28/2024, 6:37 AM"volumeMounts": {
"mountPath": "{{ mount_path }}",
"name": "{{ volume_name }}",
"subPath": "{{ sub_path }}"
}
how do i specify for when the specific flow does not need the extra mount?
also saw this https://discourse.prefect.io/t/how-to-mount-a-folder-on-my-local-filesystem-to-a-flow-run-pod-on-kubernetes/441 when i was searching. I dont want to put it into my image because the file will be huge and i dont want to download it repetitively everytime the job runs.Arnaud
02/28/2024, 11:21 AMschwannden kuo
03/04/2024, 9:41 AMLavish Sachdeva
03/04/2024, 5:38 PMSophia Ponte
03/04/2024, 10:53 PMXing Zeng
03/06/2024, 9:21 PMWilliam Jamir
03/12/2024, 4:46 PMFile "/usr/local/lib/python3.11/site-packages/prefect/runner/submit.py", line 162, in submit_to_runner
raise RuntimeError(
RuntimeError: Failed to connect to the `Runner` webserver. Ensure that the server is running and reachable. You can run the webserver either by starting your `serve` process with `webserver=True`, or by setting `PREFECT_RUNNER_SERVER_ENABLE=True`.
Tom
03/15/2024, 11:57 AMLielA
03/15/2024, 5:32 PM