Prefect Community #ask-community

raj

11/05/2025, 12:00 PM

@Marvin ❗ Issue Summary: The outage was caused by an unexpected use of an HTTP/2 protocol feature (
RECV_PING
) on a closed connection, which is invalid according to the HTTP/2 protocol specification. At the core, the *`httpx` client*—used internally by *Prefect's Cloud API client*—attempted to send a PING frame (

RECV_PING

) on an HTTP/2 connection that had already transitioned to a CLOSED
state. This caused the underlying httpcore
transport layer to raise a

LocalProtocolError

due to the mismatch in expected connection state. 🧾 Root Cause Analysis (RCA): • The system attempted to reuse an HTTP/2 connection that had already been closed, leading to an invalid protocol state. • This is most likely due to a stale connection being kept in the async connection pool or a bug in the connection lifecycle management by

httpcore

. • Prefect’s cloud client attempted communication over this dead connection, resulting in failure.

Øyvind Monsen

11/05/2025, 3:34 PM

@Marvin we are strugling a lot with error renewing concurrency release for longer running flows. It typically occurs after a somewhat heavy task involving both some CPU bound stuff with lots of async IO. We are using the default task runner, and mostly just awaiting an asyncio gather for several tasks. We updated the PREFECT_RUNNER_HEARTBEAT_FREQUENCY=30 and I can confirm that even during the longer task it is sending frequent heartbeats. However after the long task, consistently we get error renewing the lease when it switches to a new task. For context we are on prefect 3.4.25 now (have tested several versions), and we have a global limit on the deployment (to avoid hitting resource limits on number of containers) We run deployments on azure container instances. They have 1CPU available each

Arthur Ren

11/05/2025, 5:23 PM

@Marvin can you give me a example of using .deploy() method to deploy a flow targeting a prefect k8 worker with customization on the k8 job template

Slackbot

11/06/2025, 11:34 AM

This message was deleted.

Sergio Luceno

11/06/2025, 11:38 AM

@Marvin The problem: We are running server side prefect with k8s oficial helm charts. the problem we are trying to solve is the following: • Our flows are simple, they start, perform http request (can take long to respond) and stop. • We are running kubernetes workers, so we create a new pod for every execution. Every pod consumes arround 300MB of RAM • In other words, we are currently using prefect to run background jobs that we could run with Celery • It's quite expensive to us if every job, takes 300MB of RAM. What we have looked into: We've looked at alternatives, like using prefect background tasks: https://docs.prefect.io/v3/advanced/background-tasks But we are missing primitives to tell concurrency and so on to background tasks. Our new proposal: Instead of using a kubernetes worker, just use a process worker for this workload. This worker will run flows as subprocesses, we can scale the worker by CPU/RAM if that's a problem. We can benefit from deployment definitions, concurrency, and all the things. I have the impression this is a better solution for our specific case. In the future, if we have a different workload that we want to execute one flow in one kubernetes pod, we can create a new kubernetes worker for it. Is this solution an antipattern? are we doing something wrong? It seems process worker pools are meant for development or try things in local

Vít Brunner

11/06/2025, 1:34 PM

Hi, I'm a newb, using Prefect 3.4.24, self-hosting the server. While task caching works, for a task whose result has been taken from the cache, I can't quite say where the result is coming from: in the UI, the "Cache Key" in the "task-run" details is always empty (for both "Completed" tasks that have created a cache entry and "Cached" tasks which were taken from the cache). Halp please? (@Marvin?)

Arthur Ren

11/07/2025, 2:34 AM

I’m trying to setup a basic hello-world prefect docker worker but when worker spin up the container it throws a 401 error inside the container as it attempt to call /api/flow_runs/{flow_run_uuid}, I confirmed the worker process the sufficient credential to call prefect server but why can’t the container (@Marvin)

Kiran

11/07/2025, 5:49 AM

@Marvin how long data does the prefect-promethus exporter store in itssanapshot

Pierre L

11/07/2025, 9:41 AM

Hi @Marvin I am considering using the dragonfly k8s operator as a replacement for bitnamilegacy/redis in Prefect server OSS. Has it ever been tested ? Will it work ?

Giacomo Chiarella

11/07/2025, 2:26 PM

Hi everyone! I have a deprecation warning The state property of PrefectFuture is deprecated and will be removed in a future release. If you are subclassing PrefectFuture, please implement the state property in your subclass or subclass PrefectTaskRunFuture instead. what I am doing is

my_task = task.submit()

wait for it to finish and

my_task.state

how should I change the last statement to solve the warning? Same question related to my_task.task_run_id

Amrit Amar

11/07/2025, 4:56 PM

Hi everyone! My team and I are looking into using Prefect to manage our workflows on AWS. I saw the

prefect-aws

library and I'm excited to start using it, however most of the patterns for running flows seem to evolve around using another EC2 instance. I was wondering if it was possible to have a Prefect server as an orchestrator (on Fargate/EC2) that would run defined flows and deploy a 'task' on a lambda (I would have already configured the lambda, just need to send a task to it and then retrieve results to pass onto the next task which would be another lambda call). I see that lambda invoke that can happen 'within' a task but is it possible deploy the entire task as a lambda itself rather than invoking it from the main prefect instance? Also, will the workers, if invoking a lambda from the main instance, actually wait the full 15 minutes for a response from the lambda or do I need to poll the status of the lambda job with another task? And is this scalable to having 1 prefect server instance with 4 defined flows having 10s-100s of concurrent flows running? My main use case is to have an orchestrator that, upon running a flow with defined tasks, will send info to a lambda, and then get the result from it to pass it on to the next task until the end of a flow. Thank you! (@Marvin)

Quinton Nickum

11/07/2025, 9:46 PM

Hi all, Wondering how assets scale for large numbers of files - simplifying a ton here, but my team has multiple separate files in S3 (think about 10 or so files some as images) that represent different components of a single piece of data at different stages (identified by a single uuid e.g. uuid-a/file1, uuid-a/file2.......). Anyone implement something like this before? Are there limits on the number of assets Prefect is able to track? Standard ways to handle cases like this? UI limitations in Prefect cloud? Scaling concerns with events being emitted?

Aaron

11/08/2025, 11:56 AM

Does this dev/prod setup make sense? (I’ve no experience setting something like this up and limited prefect experience). We're using Prefect Server hosted on a VM. 1. Two VMs in GCP, each with their own Prefect server instance running (one for Prod, one for Dev). 2. On each VM's prefect instance, the variable "Environment" is set. For the dev VM instance, it will be set to "DEV" and in prod, "PROD" 3. Each pipeline we have will read in this variable at the start, and there will be some conditional logic basically that will decide what target tables, databases, etc to look at based on whether it's DEV or PROD. We'll have it read from the dev prefect instance when developing on our local machines by setting PREFECT_API_URL env variable on to the correct api endpoint of the dev instance. 4. Developers work on feature branches, test locally, and then merge into a development branch in Github. A Github actions job runs our deployment.py script (where all our deployments are written) to deploy to prefect server (setting PREFECT_API_URL environment variable appropriately before running the script) 5. When development branch is merged into main, a second Github actions script runs which does the same as the first, except this time it changes PREFECT_API_URL to the prod instance of prefect, so that when the deployments script runs, everything gets deployed there. EDIT: actually I may only need one branch in GitHub. Once a feature branch merges into main, the GitHub actions script deploys to both dev & prod

Riya Sinha

11/08/2025, 7:33 PM

sorry, is anyone else having trouble launching subflows when running on prefect managed infrastructure / prefect cloud? i have an hourly flow that launches some subflows, and that used to work, but for the past 12 hours the prefect python api is giving an http read error when trying to launch the subflows eg:

Copy code

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 1079, in create_flow_run
    return await client.create_flow_run(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 568, in create_flow_run
    response = await self.request("POST", "/flow_runs/", json=flow_run_create_json)

full trace in thread to avoid large message — just wanted to ask if this is an issue others are currently facing recently

11/10/2025, 7:33 AM

hi, when I try pytest prefect, it shows

Copy code

[error] tests\conftest.py:76: in <module>
    from .fixtures.api import *
tests\fixtures\api.py:12: in <module>
    from prefect.settings import PREFECT_SERVER_DOCKET_NAME, temporary_settings
E   ImportError: cannot import name 'PREFECT_SERVER_DOCKET_NAME' from 'prefect.settings' (...\Lib\site-packages\prefect\settings\__init__.py)

I don't know how settings like PREFECT_SERVER_DOCKET_NAME works, anybody help?

Nathan Low

11/10/2025, 2:41 PM

Hi All, Has anyone else been able to get github and prefect-cloud integration to work? I do the following and get an error when pip has finished installing:

Copy code

pull:
    - prefect.deployments.steps.run_shell_script:
        id: get-github-token
        script: uv tool run prefect-cloud github token owner/repository
    - prefect.deployments.steps.git_clone:
        id: git-clone
        repository: <https://x-access-token>:{{ get-github-token.stdout }}@github.com/owner/repository.git

Then I get this error when running the run_shell_script step when the flow start to run:

Copy code

`run_shell_script` failed with error code 1: Downloading pygments (1.2MiB)
Downloading uv (20.4MiB)
Downloading pydantic-core (2.0MiB)
 Downloading pydantic-core
 Downloading uv
 Downloading pygments
Installed 35 packages in 126ms
Bytecode compiled 1004 files in 1.11s

Michel Lalancette

11/11/2025, 12:20 AM

Is there a Prefect roadmap available? I'm curious to know what's next :)

Kevin Hu

11/11/2025, 7:16 PM

Hi is there a way to define a deployment in prefect.yaml that just runs a shell script without defining the flow in python? im using prefect 3.

Theom

11/12/2025, 9:41 AM

Hi everyone, I use local storage for result persistence. I'd like to set different paths for different workflows that are deployed together (through the

.to_deployment

serve

scheme). If I set the environment variable

PREFECT_RESULTS_LOCAL_STORAGE_PATH

, it's set to the same value for all workflows. I asked Marvin and I was told to set

result_storage

in the flow decorator, either to

LocalFileSystem(basepath=path1)

Path(path1)

but it fails. Somehow I need to call

.save()

on the

LocalFileSystem

instance beforehand to create a block server-side. I find this a bit convoluted for a fairly simple need, especially since I otherwise don't need to do it when using the

PREFECT_RESULTS_LOCAL_STORAGE_PATH

variable set to arbitrary values. Marvin also told me that I could pass

PREFECT_RESULTS_LOCAL_STORAGE_PATH

as a job variable to the deployment, but that it was not the recommended solution. What would you recommend? Also, if there are no other solutions than what I described, I would argue that being allowed to simply provide a

Path

object to

result_storage

would be a reasonable behaviour to expect and I would be interested in discussing it or filing an issue on GitHub. Thanks!

Phillip Shearin

11/12/2025, 3:11 PM

Hi All. I am one of the thousands of newly unemployed data analysts/engineers, and I am looking to continue my upskilling to include Prefect, while also finding an online public dataset that is continuously updated that I can do batch, or micro-batch extracts from to populate an iceberg-based data lakehouse, while also applying dbt to the workflow to build various analytics views in PowerBI, Tableau, and Apache Superset. This is all to create a demonstratable portfolio in github or gitlab CI/CD. Before I spend hours looking for that right candidate dataset, could any of you already suggest a public dataset, with a URL? Thanks in advance.

🎉 2

good luck 1

Tri

11/13/2025, 4:54 PM

Hi, Prefect Assets is currently only for cloud version, is there any plan to have it on open-source version? That's a very cool feature.

hambone johnston

11/13/2025, 8:15 PM

yo do you guys have problems w/ prefect crashing on high task loads? i've been trying to make prefect work for us for a number of months, but getting a bit discouraged: whenever a flow queues up thousands of tasks (on the order of 10,000+) prefect just outright crashes i'm using redis as the message broker, have the background services spun up separately, etc. i've resorted to writing my own very basic task queue, which works --- but given that such a simple (but important) piece of architecture solves the problem, i'm assuming i'm doing something terribly wrong? [basically: "don't queue up more than a couple-hundred tasks on prefect. wait until there are fewer than 100, then queue up 300"] but that's the kind of thing i'd want prefect itself to deal with

hambone johnston

11/13/2025, 8:16 PM

it's been kind of a struggle 😞

alex joyce

11/14/2025, 6:34 AM

Hi Everyone, I am using prefect for some automations. .I am using same configurations in both staging and production for deployment .but in staging if I make any change ,the deployment is taking the new code .but in production it is not happening. Anyone knows what will be the issue

Nathan Low

11/14/2025, 2:57 PM

Is there any way to disable the Oops. something went wrong pop ups on prefect 3? I’m worried staff against using prefect where I work will use that as a reason for the company to stop using it completely. Seems to mostly be caused by database uniqueness errors in Postgres and the database being locked on development machines.

Jordan Jones

11/14/2025, 4:18 PM

Is there anywhere I can get the prefect icon for my slack webhook integration?

Sergio Luceno

11/14/2025, 5:17 PM

Hi everyone, does anyone have faced the following? We have a self hosted prefect with EKS and we do use the official prefect helm charts In order to safe resources, and because our flows are really simple... instead of running workers kubernetes types, we switch them to run process types. After few days running this way, we noticed the following: • We have runs indefinetly in RUNNING status. They never finish. It happens because we have autoscaling, and whenever we downscale, the worker pod receives the sigterm, we cal clearly see two log lines about SIGTERM, but it stops immediately. It''s like the worker process receives the sigterm, but it does not wait to anything, just stops, so it does not gracefully stops. The deployment has a terminationGracePeriod defined of 60sec, but it does not matter as the pod automatically shutsdown without waiting. We tried to upgrade to the latest version of the image the helm chart could use ( prefecthq/prefect:3.6.1-python3.11-kubernetes) but we are having problems too cc @Marvin

Kevin Hu

11/14/2025, 5:51 PM

Hi everyone, we experienced a bug where after re-deploying a flow, the prefect server created duplicate scheduled runs of the flow. when we manually disabled the schedule and re-enabled it, the server correctly only generated single runs. is this a known issue with updating existing deployments?

Chu

11/14/2025, 7:42 PM

https://prefect-community.slack.com/archives/C04DZJC94DC/p1763146961194199?thread_ts=1763145223.026799&cid=C04DZJC94DC Is this right that a subflow inherits the

result_storage

from the parent flow, even if the subflow had its own

result_storage

set?

Sơn Lê

11/15/2025, 10:01 AM

Hi everyone, i'm trying to build a pipeline which have the following components: • Schedule Coordinator: Receive event through a queue and parse the event for scheduling job. We are using github storage to create the flow (to avoid import many dependencies of different workflow into scheduler). • A workpool in K8s • Worker running flow in K8s also Everything is working but we have a problem with the Scheduler Coordinator caused OOM error on K8s even we have tried to increase the memory for the pod, the problem is happened when the schedule coordinator trying to pull github (our github repo size is only 5 MB and we already increase the memory of the pod to 1Gb). Is anyone have this problem before? And beside using github storage to create a flow, is there any different way to create flow without needing to import dependencies of these flows? Thanks!