hey, I'm getting server errors under heavy loads when self hosting on k8s

sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: <https://sqlalche.me/e/20/3o7r>)

this causes flows to crash and sometimes creates zombie flows that I have to cancel/delete maually the loads are around 100 flows (each flow has around 100 sub-tasks), and flows are running on Process type agents I can increase these settings on the server, or maybe even put in a better Postgres DB, but i'm not sure how to avoid these errors in the future

Hey All,

...

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: test_flow
  entrypoint: flows/test_flow.py:test_flow
  work_pool:
    name: dimensional-etl

I added GCP push as my architecture (work pool) with above given prefect.yml. But when trying to trigger the flow run I am getting below error. Do you have any idea what's happening? Many thanks

Failed due to a(n) InfrastructureError Flow run infrastructure exited with non-zero status code: Exited with non 0 code. (Error Code: 1). This may be caused by attempting to run an image with a misspecified platform or architecture.

Steps I took to get to this point are: 1. Add a GCP cloud run push as work pool named "dimensional-etl" 2. Create a test flow 3. Add entry in deployments section in prefect.yml file 4. Github action to deploy flows 5. Then trigger a flow run from UI Am I missing something? @Jenny

Hi prefect community ! I'm actually using version 2.14.9 and setting my environment variables as you can see in the image below but it is not working, as for example, I'am trying to activate the debug mode for the logs but nothing change

Hello, has anyone found a way to use Azure Workload Identity on AKS when connecting the prefect-server to a postgres DB?

Hi all does anyone know of a way to specify an ACL when putting an object in s3 using the prefect.filesystem.S3.put_directory() function?

Hello, Having issues getting k8s flow run to pull custom image from our dockerhub repo. Getting "Failed to pull image "myrepo/myimage:dev-prefect2": rpc error: code = Unknown desc = failed to pull and unpack image" error. Secret for our docker hub repo is a k8s secret. In prefect 1 I would set the run_config=KubernetesRun(image="myrepo/myimage:0.0.44", image_pull_secrets=['dockercloud-secret'], cpu_request=4, memory_request=16, env=env_var), How should I be doing this in prefect 2? Where should it go? Kind of confused who needs the secret. In my head the worker should know how to pull the image. I have tried setting it in the deployment like the following based on another post but is still failing. deployments: - name: dev version: null tags: [] description: null entrypoint: my_flow.py:my_flow parameters: {} work_pool: name: dev-workpool work_queue_name: null image: myrepo/myimage:dev-prefect2 job_variables: {image_pull_secrets: ['dockercloud-secret']}

Has anyone had any success doing an image build (leading up to a deploy) on macOS Sonoma; M2 CPU; using colima? I'm seeing a few errors during the Docker image build step that point at Prefect not being aware of how to find the colima equivalents to what vanilla Docker provides. The errors are below. (1) lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect sock.connect(self.unix_socket) FileNotFoundError: [Errno 2] No such file or directory (2) lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect sock.connect(self.unix_socket) urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')) (3) lib/python3.11/site-packages/requests/adapters.py", line 501, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')) (4) lib/python3.11/site-packages/docker/api/client.py", line 221, in _retrieve_server_version raise DockerException( docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')) (5) lib/python3.11/site-packages/prefect/deployments/steps/core.py", line 182, in run_steps raise StepExecutionError(f"Encountered error while running {fqn}") from exc prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_docker.deployments.steps.build_docker_image

In a prefect job run, could I get my job run url programmatially? We wish to send a report containing this job run url, so that people would be able to go to the original job run log.

Hey does anyone came across below error.

Validation of flow parameters failed with error: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 434, in retrieve_flow_then_begin_flow_run
    parameters = flow.validate_parameters(flow_run.parameters)
  File "/usr/local/lib/python3.9/site-packages/prefect/flows.py", line 502, in validate_parameters
    validated_fn = V2ValidatedFunction(
  File "/usr/local/lib/python3.9/site-packages/pydantic/v1/decorator.py", line 126, in __init__
  File "/usr/local/lib/python3.9/site-packages/prefect/_internal/pydantic/v2_validated_func.py", line 51, in create_model
    class DecoratorBaseModel(BaseModel):
  File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line 92, in __new__
  File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line 308, in inspect_namespace
  File "pydantic/fields.py", line 32, in init pydantic.fields
  File "pydantic/class_validators.py", line 8, in init pydantic.class_validators
  File "/usr/local/lib/python3.9/site-packages/pydantic/_migration.py", line 278, in wrapper
ModuleNotFoundError: No module named 'pydantic._internal._validators'

Packages: prefect==2.14.5 prefect-dbt==0.4.1

Hello, I have a small issue on Prefect Cloud using Cloud Run Push Worker. Sometimes a flow crashes because the random name of the job generated by Prefect already exist. I had this issue 4 or 5 times since I use Prefect. I know that disabling the "Keep Job After Completion" option would fix this problem, but shouldn't the job name be random enough to avoid any conflicts?

Hi all! I've implemented oauth2.0 for my Prefect UI hosted on kubernetes. However, I'm encountering an issue where the Prefect CLI is unable to establish a connection with the PREFECT_API_URL due to this authentication system. I used to authenticate with username and password, but I am not sure if prefect offers token authentication. Can someone help me with this issue?

Hey All, 2 of the prefect instances we manage with ECS started to fail all of their flow runs yesterday afternoon. Im looking at the logs and seeing this error, has anyone else seen this?

Hello 👋 I am setting up

prefect.yaml

deployments and have 2 types of pull actions I am assigning to YAML aliases (dev/prod). This is the error I get at runtime:

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 110, in run_step
    raise ValueError(
ValueError: Step has unexpected additional keys: repository, access_token, prefect.deployments.steps.git_clone

Here is my

prefect.yaml

, am I misusing aliases here?

name: my-prefect
prefect-version: 2.14.9

actions:
  pull: 
    dev: &pull_dev
      - prefect.deployments.steps.git_clone:
        repository: <https://github.com/my-repo.git>
        branch: dev
        access_token: "{{ prefect.blocks.secret.my-secret }}"
    prod: &pull_prod
      - prefect.deployments.steps.git_clone:
        repository: <https://github.com/my-repo.git>
        branch: main
        access_token: "{{ prefect.blocks.secret.my-secret }}"

build: null
push: null
pull: null

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: my-deploy
  version: 1
  tags: []
  description: 
  schedule:
  entrypoint: 
  parameters: {}
  work_pool:
    name:
    work_queue_name: 
    job_variables: {}
  pull: *pull_dev  # I want this deployment to pull from the dev branch

I am able to successfully run the deployment with a single

pull:

definition, but it breaks when trying to use an alias to define the pull action on the deployment pull:

*pull_dev

Hey Hi, Is there a way we can see the fastapi JSONResponse statements in the prefect-ui?

I'm having trouble with my

prefect.yaml

as well - it doesn't seem to be downloading the flow code:

Hi, I'm unable to create a deployment from a prefect.yaml. Am I missing something? I execute the following via CLI:

$ prefect deployment apply prefect.yaml

I get the following error:

'prefect.yaml' did not conform to deployment spec: AttributeError("'NoneType' object has no attribute 'name'")

Contents of the prefect.yaml.

name: a_flow
prefect-version: 2.14.9

pull:
- prefect.deployments.steps.set_working_directory:
    directory: /app

definitions:
  tags: &common_tags
    - development
    - eks
  work_pool: &common_work_pool
    name: development-kubernetes
    job_variables:
      image: "{{ $PREFECT_IMAGE_NAME }}:$CIRCLE_SHA1"

deployments:
- name: default
  entrypoint: main.py:run_flow
  path: /app
  tags: *common_tags
  version: "{{ $CIRCLE_SHA1 }}"
  work_pool_name: *common_work_pool

Hi, what should the ENTRYPOINT of my custom docker image (for executing runs in Kubernetes; not based on a prefect image) be? Right now I have it running my flow directly as described here: https://docs.prefect.io/latest/guides/docker/#writing-a-dockerfile but I'm getting a double flow-run - the first has the same name as my Kubernetes Pod so is presumably controlled by Prefect, but it gets stuck in "Pending" and never receives any logs from the flow. The second run has a different name than the pod and works normally (though it seems to have some problems the first time it runs on a new Node, but I'll worry about that later). Should I instead have something like

exec "$@"

as here: https://github.com/PrefectHQ/prefect/blob/main/scripts/entrypoint.sh ? Apologies if the solution is obvious/in the docs somewhere.

Question for the prefect devs -- is there anything on the roadmap to expand features around artefacts? Ability to upload / register pngs for instance?

Hello @alex @Will Raphaelson Currently we using one prefect account and we created the multiple Prefect Work Pools agents and attached with multiple server for running the flows, while registering the flow in one server it's automatically taking the default work pool agent or another server work pool agent, How to setup the work pool agent like registering the flow in test-1 server it's only used the "test-1" work pool agent.

Updated docker-compose and Dockerfile for on-prem deploys? In #prefect-getting-started when I mentioned (now archived)

docker-compose

that was top google search, it was implied to be out of date since Prefect is moving off Agents etc. Does anyone have a solid up-to-date docker-compose and Dockerfile for on-prem server (or my local laptop for dev) they could share (esp if battle tested in prod?)? The idea here being able to run community for pipelines in-docker (not worried about deployed dockerized individual flows at the moment).

(I'm assuming I need a combo of

docker-compose

and a build

Dockerfile

since I have a things like

astropy

etc I need to have available to prefect.).

Is it possible to create prefect webhook notification with json data populated by flow run url? Is there a predefined variable we could use?

When testing Prefect flows we can use the harness utility function and a fixture to ensure an SQLite database is used. This has a startup cost, and we have the overhead of actually executing our flows as a series of tasks and functions. However, we can also run

.fn

on a flow to run it directly. Unfortunately, this results in an error if the flow calls tasks that aren't also ran using

.fn

. Is there a way to run a flow such that both the flow itself and all tasks are automatically called as

task.fn

and

flow.fn

? This would be really useful for lightweight E2E tests that don't need the resilience of actually running the flows, checking states etc.

How do i change this timeout value, no

PREFECT_

is set to 30 and the

PREFECT_API_DATABASE_CONNECTION_TIMEOUT

doesn't seem to change it

sqlalchemy.exc.TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30.00 (Background on this error at: <https://sqlalche.me/e/20/3o7r>)

hi all, i need help understanding why kubernetes job doesn’t set image which i specify in code. i have latest prefect running in kubernetes, both server and worker. i create a deployment like this:

@flow(name="test")
def main():
    out = ShellOperation(commands=['echo hi'], stream_output=True).run()
    print_output(out)
    trace_success()

if __name__ == '__main__':
	Deployment.build_from_flow(
        flow=main,
        name="test",
        schedule=(CronSchedule(cron="0 0 * * *", timezone="UTC")),
        work_pool_name="default",
        infrastructure=KubernetesJob(
            image='***.<http://dkr.ecr.us-east-1.amazonaws.com/prefect:prefect-clickhouse-backup_6e7690c|dkr.ecr.us-east-1.amazonaws.com/prefect:prefect-clickhouse-backup_6e7690c>',
            namespace='prefect',
            finished_job_ttl=60*60*24*7,
            env={
                'CLICKHOUSE_HOST': 'clickhouse.clickhouse.svc.cluster.local',
            }
        ),
        tags=["clickhouse"],
    ).apply()

when i navigate to deployment i see that it uses anonymous infrastructure block: The problem is that whenever i run a deployment i don’t set image to why is specified here, also doesn’t set env. It just runs with default prefect image. What am i doing wrong?

hi prefecters. Does anybody know how to enable all boto3 logs (the underlying requests which are sent to aws) in prefect-aws for ECS worker?

Does anyone know how to override the

ephemeralStorage

parameter for ECS Tasks in Prefect Cloud? I tried adding the snippet below to the Job Variables configuration, but based on the logs it looks like each ECS Task is still using the default (20 GiB). Using this configuration works to override the default

cpu

and

memory

parameters, so it’s unclear why overriding this parameter isn’t working:

"ephemeralStorage": {
    "sizeInGiB": 40
  }

Hi! In prefect 1.X you could use Adjustments with Clocks, which was very useful for example to send an email daily (a daily report), but only for business days (excluding holidays too). What I did was to generate a clock signal daily and, with a custom adjustment, check if that day is business or not, and add X days to the datetime if it is not business day. Therefore, I receive every daily report but "delayed" those reports that are holidays to the next business day. Anyone knows how to do this using rrules? I see that this could be the only way to do that now in Prefect 2.X. Thank you in advance!

Hi all, after successfully utilizing an ecs:push workpool with my own provisioned AWS infrastructure to deploy my flow to and run it on the ecs cluster, I was curious about the new feature to let prefect provision the infrastructure 😃 Unfortunately, it doesn't work as expected. I created the ecs: push work pool as suggested (https://github.com/PrefectHQ/prefect/pull/11267) and after that I deployed my flow to it as usually, but without job-variables (since I don't have to ingest the AWS infrastructure and role arns now). I want to push the flow image to my ecr repo, so I give this with the name of the image to my Deployment like this:

flow.deploy(
    ...
    image=DeploymentImage(
        name=os.getenv("ECR_REPO_URL", ""),
        tag=os.getenv("IMAGE_TAG"),
        dockerfile=cfd / "Dockerfile",
    ),
    ...
)

But then I get the error: Flow run could not be submitted to infrastructure: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support ECR images. Why do I still have to provision the deployment with an execution role, shouldn't it (or the work pool) create one? Or is it because it is a my own ECR Repo? Where do you usually push / save the flow image to run it on the ecs cluster (especially in the frame of the ecs:push work pool with infra provisioning)? Thank you and best regards!

hey all, i'm looking at the server logs and i'm getting a ton of warnings:

2023-12-07T15:19:54+02:00 13:19:54.822 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 333.405324 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T15:24:52+02:00 13:24:52.771 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 297.947375 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T15:29:54+02:00 13:29:54.353 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 301.581939 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T15:35:23+02:00 13:35:23.989 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 329.635069 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T15:41:26+02:00 13:41:26.275 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 362.284819 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T15:46:40+02:00 13:46:40.591 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 314.314349 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T15:52:41+02:00 13:52:41.746 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 361.154281 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T15:59:18+02:00 13:59:18.214 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 396.467137 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T16:04:05+02:00 14:04:05.897 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 287.68302 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T16:09:54+02:00 14:09:54.378 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 348.479282 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T16:16:08+02:00 14:16:08.516 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 374.137448 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T16:22:20+02:00 14:22:20.136 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 371.61939 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T16:27:01+02:00 14:27:01.589 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 281.451967 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T16:32:15+02:00 14:32:15.239 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 313.649418 seconds to run, which is longer than its loop interval of 20.0 seconds.
2023-12-07T16:37:58+02:00 14:37:58.483 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 343.243348 seconds to run, which is longer than its loop interval of 20.0 seconds.

the servers are unders stress sure but this seems a little excessive i'm self hosting on k8s, and these errors happen with a local or remote (AWS RDS) DB should I be worried?