Morning, I've contacted support already, but is anyone else who uses cloud run workers seeing infrastructure errors due to prefect creating 3-4

CreateJob

requests in a row? Results in a 409 error for us, appears like so in logs:

Hello! the service started deteriorating in Prefect Cloud, for instance, we woke up to 10 crashed flows because what seems to be a rate limiting rule

Failed to generate job configuration: Client error '429 Too Many Requests' for url '<http://orion-internal:4200/api/accounts>...
For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429>

a few more also were stuck at the pending state.. is this sort of rate limiting documented somewhere for the Pro Plan? is it something we can control?

Can 'Secret' block types still be accessed in 3.6.2? I can see the secrets on the website ux, but 'prefect block ls' and python code using Secret.load('[Secret_name]') can't seem to find secrets.

Hi ? How do we set minimum resources for a prefect deployment ? I have kubernetes workers, and I need to tell them how much memory my script need to be able to autoscale my kubernetes job when deployment is running... (in Prefect 3x btw). ChatGPT give me this but I dont see any doc about it :

deployments:
- name: my_deployment
  work_pool:
    name: my_worker_pool
    job_variables:
      pod:
        spec:
          containers:
            - name: flow
              resources:
                requests:
                  cpu: "2"
                  memory: "8Gi"
                limits:
                  cpu: "4"
                  memory: "8Gi"

Hi, would someone be able to review https://github.com/PrefectHQ/prefect/pull/19498 ? Thanks!

Hey all, we're running into this scheduling issue, breaking our production pipelines. Is there an ETA on a fix here? Folks are starting to push for moving off of Prefect due to this.

hey everyone, started getting this weird issue with my Cloud Run Jobs based work pool recently. I have a flow that submits 7 flows (they are actually standalone flows with their own infra, not actually subflows). Each flow trigger its own cloud run job with a unique (prefect generated) name etc. Still I'm getting this GCP error claiming resource already exists. It seems to happen quite randomly. As you can see some of them launch successfully and sometimes all of them do. However as you can imagine it's a big problem when some of them don't. Anyone has an idea of what it can be? When I look at Cloud run I never see the job existing at any time that would justify a duplication. However it's also probably because we have

keep_jobs

set to false so even if it's duplicating some job it's probably also deleting. We're not tracking the flow state or anything like that from python, we just "fire and forget" these flows. This is my code to launch them:

for model in collection_models:
        scenarios = model_to_scenarios.get(model, [])
        if not scenarios:
            logger.warning("No scenarios found for model %s, skipping", model)
            continue

        # Submit task for this model's subflow with model-specific name
        future = launch_model_subflow_task.with_options(
            name=f"launch-{model}",
            retries=2,
            retry_delay_seconds=60,
        ).submit(
            model=model,
            scenarios=scenarios,
            collection_limit_per_brand_model=request.collection_limit_per_brand_model,
        )
        deployment_futures.append(future)

and the definition:

flow_run = await run_deployment(  # type: ignore[misc]
            name=f"process-batch-subflow/{PREFECT_DEPLOYMENT_SUFFIX}-subflow",
            parameters={
                "model": model,
                "collection_scenarios": scenarios,
                "collection_limit_per_brand_model": collection_limit_per_brand_model,
            },
            job_variables={
                "cpu": CPU_CORES,
                "memory": "32G",
            },
            timeout=0,  # Fire-and-forget: don't wait for subflow completion (execution timeout is set on the subflow itself)
            tags=[model],
        )

@Nate, any chance you can approve the auto-bot updates for conda

pydocket-feedstock

to unblock conda

prefect-feedstock

? This will help us update our env, which would be nice! https://github.com/conda-forge/pydocket-feedstock/pulls

hi all, question about the

ProcessPoolTaskExecutor

(or really, any of the parallel task runners). Since tasks can be nested inside of tasks, what's the execution model when i

task.submit()

a task, and it calls

another_task()

inside of it, where does

another_task

run? in the process that is handling

task

? i ask because it used to be that we needed to wrap tasks in flow runs to control which task runner to use

Hey all, we're seeing weird behavior with Prefect Cloud jobs when a single flow run is picked up by two ECS tasks and both execute the flow simultaneously. The logs in Prefect Cloud then appear duplicated, and one of the ECS tasks eventually fails with exception

prefect.exceptions.MissingResult: The result was not persisted and is no longer available

We observed this issue three times, first one on Saturday, then Monday, and today. This is the first time we encountered this issue after using Prefect Cloud for over 2 years now. We're using ECS Push work pools. Does anyone experience the same issue?

Hello! I'm self-hosting Prefect with SQLite and I am getting a lot of these errors: https://gist.github.com/valleybay/bb1bd9717596826e125b0343fb20c64a. It says «database is locked». It's repeating every few minutes. Anyone else experiencing this? Have you been able to get Prefect to run smoothly on SQLite?

Hi @Marvin I am hosting Prefect OSS v3.6.3 on kubernetes. I have an automation that send a slack alert when a flow enters TimedOut, Crashed, or Failed. Yesterday I got some alerts in slack, it was working. Now when a flow crashes, nothing arrives on slack even though the automation is turned on. The slack block is working: when I run these lines locally, I get the alert:

>>> from prefect.blocks.notifications import SlackWebhook
>>> slack_webhook_block = SlackWebhook.load("slack-prefect-prod-failures-v2")
>>> slack_webhook_block.notify("Hello from Prefect!")

I create the automation locally with

prefect automation create --from-file automations.yaml

. Creating the simplest automation in the UI doesn't solve the problem. What could be the cause ? I did not changed much things since yesterday.

Hi. What might cause the Prefect UI to not display task logs in the task log view, and instead display:

This run didn't generate Logs

Task logs exist in the DB and do display correctly in the flow log view. The task logs used to display at the task log view until about 2 days ago. It may have something to do with a single new flow which was added which called

get_run_logger

, a pattern which did not exist previously, has since been removed, but the problem remains globally for all our flows.

Trying to get this PR across the line: https://github.com/PrefectHQ/prefect/actions/runs/19767933876/job/56645184779?pr=19559 But I'm seeing CI fail on python3.10, python3.12 (passes for 11, 13, 14). I suspect flakiness could be a problem since there shouldn't really be differences exposed by this change in terms of python version (just adds a setting). Is this a known problem with the Github CI suite for Prefect?

Hi @Marvin, how to use environment variables placeholder in prefect.toml?

@Marvin What is the difference between call task.delay() and wrap a task into a flow? My senario is invoking long running logic through fastapi, thus I want to query the task state, task log, task result etc through my frontend.

@Marvin What is the difference between "calling a task directly in a flow" and "calling task.submit() within a flow"?

Hello everyone, I am running Prefect on my local machine. I am using

task.submit()

followed by

task.result()

to retrieve the output. However, even a simple Python function (e.g., retrieving a value from a dictionary) takes around 30 seconds to complete.

#CL09KU1K7 #C04DZJC94DC

Hi folks 👋 on my on-prem setup with kubernetes work pool, we’re trying to use OpenTelemetry in the flow runs, sinking the telemetry data to a Otel Collector, but we’ve observed two things:

OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true

does not work with formatting the log output. Adjusting

PREFECT_FORMATTERS_*

leads to missing otel’s injected properties. traces and metrics set inside the flow code does not propagate to the Otel collector is there a magic trick to have Otel working with Prefect?

Hello in my AWS worker I am seeing an error: ``fatal: not a git repository (or any of the parent directories): .git` I have tried many things ( my gh PAT is correct, my repo is valid, the branch exists ). Can anyone help please?

Hi, what is the eviction policy of task cache (like LRU etc)? Thanks

Hey all - I have a question about how to do something in Prefect. I want to kick off a particular flow once 1) we read a request for some work from a queue in the cloud AND 2) a certain set of flow dependencies have succeeded. Currently, the way we achieve this is by polling for success of the dependent flows in the flow triggered after reading the request from the queue. This isn't great, obviously, since it wastes workers and doesn't proceed as fast as it should once dependent flows complete. Any recommendations about how to do this more elegantly in Prefect?

hi yall, i am trying to use prefect and logfire/instrumenting logfire correctly, but have been running into a lot of issues with orphaned spans (we try to wrap each prefect flow/task in its own span for easier to read logs). would love to know if anyone has experience using the prefect/logfire and has tips

Hi 👋 I signed up for Prefect Cloud yesterday and so far I haven’t been able to use it at all. • Nothing is loading properly • I’m getting 404 errors on my flow pages • And when I log in / sign up, I get stuck on a loading screen (see attached screenshot) Is this a known issue, or has anyone experienced this before? Thanks

hey everyone! We’re on Prefect Cloud with ECS workers (Prefect 3.6.5, prefect-aws 0.7.1). ECS worker runs on prefect-ecs-cpu-cluster (Fargate). infrastructure_pid is set on runs (e.g., prefect-ecs-cpu-cluster:arnawsecsus-east-2:.../task/...). IAM for the worker task role includes ecs:StopTask. Region/queue are set (AWS_REGION/AWS_DEFAULT_REGION=us-east-2, PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_SQS_QUEUE_NAME=prefect-ecs-events). SQS/EventBridge queues are empty (no backlog). Problem: Cancelling a flow run leaves it in “Cancelling”; the ECS task keeps running. Worker logs show no StopTask or cancel handling. Even with infrastructure_pid present, no stop is issued. Forcing state via UI/CLI doesn’t stop the task. Any known issues or fixes to get Cancel to issue ECS StopTask? Should we tweak anything on the worker side, or is there a known bug in the cancel path?

Hi everyone, Is there any documentation available for hosting prefect open source in azure container apps?

Hi folks. Do you if there's a good book about the version. I'm trying to convince my airflow-fan of coworker to take a look to Prefect, but I need to be strong myself about this. Thanks

Hi. we often get a "504 Gateway Time-out" when Prefect tries to pull a docker image from Docker.io. We are using Prefect Cloud + ECS Push workpools. Any ideas in which direction we should investigate?

Hi @Marvin I run prefect OSS v3.6.3 on a managed k8s cluster, connected to CNPG/cloudnative-pg /cloud native postgresql. All pods have 2 replicas spread on 2 nodes. I sometimes have flow runs crashing with :

prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://prefect-server.prefectoss.svc.cluster.local:4200/api/flow_runs/dbdf008d-f9d3-42b1-be21-ef2a4b12b567>'

(url can change) I found these logs in the prefect server pod :

11:08:48.406 | ERROR | prefect.server - Encountered exception in request: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/asyncpg/connection.py", line 2421, in connect return await connect_utils._connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 1049, in _connect conn = await _connect_addr( ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 886, in _connect_addr return await __connect_addr(params, True, *args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 931, in __connect_addr tr, pr = await connector ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 818, in _create_ssl_connection new_tr = await loop.start_tls( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1268, in start_tls await waiter asyncio.exceptions.CancelledError

In the logs of postgres, at the same time, I have :

{"level":"info","ts":"2025-12-10T11:08:48.310120701Z","logger":"postgres","msg":"record","logging_pod":"cnpg-database-cluster-2","record":{"log_time":"2025-12-10 11:08:48.306 UTC","process_id":"386195","connection_from":"10.2.14.94:42384","session_id":"693954b9.5e493","session_line_num":"1","session_start_time":"2025-12-10 11:08:41 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"08P01","message":"SSL error: unexpected eof while reading","backend_type":"not initialized","query_id":"0"}} {"level":"info","ts":"2025-12-10T11:08:48.31023509Z","logger":"postgres","msg":"record","logging_pod":"cnpg-database-cluster-2","record":{"log_time":"2025-12-10 11:08:48.307 UTC","process_id":"386195","connection_from":"10.2.14.94:42384","session_id":"693954b9.5e493","session_line_num":"2","session_start_time":"2025-12-10 11:08:41 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"08006","message":"could not receive data from client: Connection reset by peer","backend_type":"not initialized","query_id":"0"}} {"level":"info","ts":"2025-12-10T11:08:48.310409409Z","logger":"postgres","msg":"record","logging_pod":"cnpg-database-cluster-2","record":{"log_time":"2025-12-10 11:08:48.309 UTC","process_id":"386196","connection_from":"10.2.14.94:42388","session_id":"693954b9.5e494","session_line_num":"1","session_start_time":"2025-12-10 11:08:41 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"08006","message":"could not accept SSL connection: Connection reset by peer","backend_type":"not initialized","query_id":"0"}}

The prefect-server pods have not restarted and they do not seem limited by resources. we have :

Limits: cpu: 300m memory: 600Mi
Requests: cpu: 300m memory: 600Mi

while the max cpu usage I see on grafana peaks to 82m. My questions : knowing that all is working most of the time, do I need to set

PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED=true

? If not, explain if I should set these environment variables and which values I should set: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE PREFECT_SQLALCHEMY_POOL_SIZE PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT