I am running into an issue where Deployment Runs are stuck in AwaitingConcurrencySlot even though nothing else is running. I have Concurrency set to 1, and it has always worked as intended before. If I set the Concurrency to 2, they start working again, but that doesn't explain why it thinks something else is already running. Any thoughts?

Hello prefect team, I am having a lot of trouble with one of our ecs push work pools. I commented on a related issue here: https://github.com/PrefectHQ/prefect/issues/18429#issuecomment-3263294213 Each night, we have a wave of ~300 runs get submitted. They all end up running many hours late because all the runs sit in LATE status. The concurrency limit on the work pool is 20, and the deployments don't have concurrency limits. I don't think it is an issue with ECS infra, because I can't see any logs, it looks like the runs are never being submitted at all. But eventually, they do. It seems like maybe something on the prefect backend is telling cloud not to submit them? But I am not sure where to look to get this information if it is something on our ECS side that could be causing this.

Hey @Marvin prefect.flow_runs Finished in state AwaitingRetry('FLOW_FAILURE:OOM:Process exited with code 137 — SIGKILL detected (likely due to Out Of Memory or external kill).', type=SCHEDULED) , prefect.exceptions.UnfinishedRun: Run is in SCHEDULED state, its result is not available. thsi is from my flow run log, Iam failing my flow with return Failed(('FLOW_FAILURE:OOM:Process exited with code 137 — SIGKILL detected (likely due to Out Of Memory or external kill). upon hitting a conidtion, and i have added retries , why am i seeinga awaiting retry and unfinished run errror

Hey Prefect team, it looks like the implementation for setting an environment_name job variable for a Modal pool was broken by recent Modal updates. You can no longer pass environment_name as an argument during Sandbox creation (deprecated in Modal 1.1.0); this causes any Prefect job with this variable set to crash immediately upon submission. Could you please fix this or remove that job_variable from the template? FWIW, this unexpectedly broke all of my Modal flows; would be great if you all could keep a closer eye on Modal deprecations now that they are in the 1.0+ state; since your Modal submission code is closed source, it is hard to fix/debug on my end.

Our `Manage subscription`button under settings/billing is giving a general error message - is it just me?

hey @Marvin Iam running my prefect flows by creating the deployments on ecs using faragate spot , i see that iam getting my deployment id ,name, flow name and flow run id on my ecs tasks as tags in the aws console, how are they getting generated

@Marvin how do I set extra pip packages in a managed deployment in the Flow.deploy() function in a python sdk?

@Marvin all my flows are late, enter in Pending state and then crash with "Flow run infrastructure exited with non-zero status code: Essential container in task exited (Error Code: 1)", I'm using the cloud's managed work pool and execution infrastructure

Hi all, I'm trying to use the

GitRepository

source for my flow deployments but now realise my mistake as I need to both

git clone

and

pip install

my package. What's the correct way to clone and install my deployments? The docs don't provide examples for this sort of flow.

How can I completely block these Prefect logs? I did not opt in to this. Seems to be a behavior change from 2.x to 3.x

Finished in state Completed(message=None, type=COMPLETED, result=ResultRecord(metadata=ResultRecordMetadata(...

Every time one of my tasks finishes, its outputs are involuntarily printed to the cloud logs. This is unacceptable as some task outputs contain secrets such as auth tokens, and now we have to deal with leaked secrets.

Am I the only one having a hard time editing my work-pools? They seem to only crash with I load the edit workflow page on the browser. I'm using Safari and Brave and I'm getting the problem on both. When I use safari it just loads slowly and doesn't let me complete the edited page and on Brave it just crash.

Im constantly getting this message despite having a version higher than that mentioned.

hi @Marvin , I am calling some sub flows (another deployment)from a main flow/deployment, the main deployment takes some tasks from a config file and calls the subflow/sub-deployment by iterating over each task, i am triggering 748 tasks, so 748 sub flows , I see that more than 748 subflows getting triggered and i see that that for some task names more than 1 subflow is getting triggered, can you explain this behavior?? is it something that is observed before?

hi @Marvin Sep 3rd, 2025 Worker 'ECSWorker 7b242ba6-2b7d-4a2c-a8b7-0acb33b775b7' submitting flow run '18e89d5d-0b60-410d-ae45-6eb4f60a055e' 062743 PM prefect.flow_runs.worker Retrieving ECS task definition 'arnawsecsregionaccountnumtask definition/somegroup 0131'... 064811 PM prefect.flow_runs.worker Ignoring task definition in configuration since task definition ARN is provided on the task run request. 064811 PM prefect.flow_runs.worker Using ECS task definition 'arnawsecsaf south regionaccountnumtask definition/somegroup 0131'... 064811 PM01:31'... 064811 PM prefect.flow_runs.worker Creating ECS task run... 064812 PM prefect.flow_runs.worker Waiting for ECS task run to start... 064813 PM prefect.flow_runs.worker ECS task status is PROVISIONING. 064813 PM prefect.flow_runs.worker ECS task status is PENDING. 064834 PM prefect.flow_runs.worker ECS task status is RUNNING. 064904 PM prefect.flow_runs.worker this is from one of my prefect flow run log , here the worker is submitting the run at 6:27, but the task is getting created at 6:48, why ? is it becuse iam submitting too may runs

Hi ~ Is there any way to make the cached task_run show the artifacts of the original task_run on the web UI?

Hi @Marvin I'm using background tasks to trigger flows (with sub-flows and tasks / sub-tasks) that might take some time to finish. I have multiple replicas of the backend calling the background tasks as well as multiple replicas of the workers. I must make the system resilient to restarts (no-downtime deployment, potential crash of workers, etc.). The flow is idempotent so it can be restarted by another worker after the original one would have been stopped. => So I need a way to detect flows that are not attached to a worker anymore and "reschedule" them accordingly. What would be the appropriate way to achieve that ? I saw this page talking about zombie flows detection but my flows are still stuck in running after I restart my worker container. Asked Marvin and it told me the flow would move to "crashed" state, while it stays "running" forever for me. (Using Prefect 3.4.17) Thanks ! 🙂

@Marvin I deployed a self hosted prefect on google cloud kubernetes. The server cant be setup and I got serveral postgres errors: prefect-server-postgresql-connection

@Marvin we are running a prefect deployment in k8s and are hitting issues when going over 200 parallel runs. Running latest version 3.4.17. also tried 3.4.1. We are using a postgres database. When adding pgbouncer we configured the pool size to 200 and 300. Hitting sqlalchemy timeouts. After that we tried increasing the pool size to 2000. Increased the pool size on pgbouncer to 3000. Same issues. We are seeing around 700 connections. We tried connecting directly to the db. 300 parallel jobs ran. Going beyond 1000 we ran into maxing out our db, not a small one - 64gb me more and 8 vcpu What is going on?

@Marvin I tried setting up a block using the Custom Webhook type, when I click the 'Create' button, none of the parameters show up in the UI. Any suggestions?

how to change my profile email in prefect?

hello there! Does anyone know if there's a final state to signal that a given workflow has warnings? Currently there's essentially

Failed

or

Success

https://docs.prefect.io/v3/concepts/states#states I need for my data pipelines to be able to signal warnings, but I don't have a proper way to signal that via the DAG final state; which would be ideal for observability. Is there anything to support 3 final workflow states, like a semaphore: •

Failed

🔴 •

Warnings

/

Success_with_warnings

🟡 •

Success

🟢 Or any other monitoring solution within the

UI

or observability side of

Prefect

that allows that? CC: @Nathan Nowack @Brendan O'Leary @Anna M Geller

Are there any information about self-hosting an enterprise version of Prefect Cloud? I'm evaluating Prefect for our company (Infineon) and on-prem is a hard requirement. Any information about how the service and pricing models is designed in this case?

Prefect appears to be hanging on task shutdown (see attached) waiting for the tstate (thread state?) lock to do some telemetry work. Task has been going for 30 min, has a timeout of 60s, and has not been retried per the prefect UI. Is there something I might have done to cause this? Any tips to troubleshoot further?

@Marvin my flow runs keep crashing. I do see some error in the runs but it should be going to failed state and not crashing state. I am on prefect 3.1.15

Hi everyone! I’m using Prefect 3.2.14. When I click on Retry button of a failed flow run it is never restarted. It remains in AwaitingRetry. Why is that? I’m using Process infrastructure.

Hi all - I have a set of perfect version 2 workflows that I am trying to get to work under the version 3 engine. I am using a DaskTaskRunner., and have very simply

@task

decorated functions without any asyncio. I am seeing that all the tasks are now returning

prefect_dask.task_runners.PrefectDaskFuture

. Oddly they are not being used to create any tasks, or general workflow. Each time I run I am not seeing any dask tasks get registered in the web UI. What is confusing my is that I am seeing the logs in the version 3 web UI. Is there now some differencce in how a dask task running is meant to be used? Is it automatically started and in a state ready to receive work? Or am I meant to be resolving these futures explicitly before they are added to the dask work graph?

@Marvin how can I pass params to a flow

deploy()

function in the Python SDK?

Hello, I've noticed that in my flow runs, there are tasks that show completed, but the task continues to run. The flow also finishes in a completed state. These are run locally for now, but I want to ensure that when I convert these to a deployment they do not exhaust my compute credits, if we choose to go that route. Here are the params for the task decorator: @task(name="Validate Data", timeout_seconds=60, persist_result=False)

Hi, I'm running into some issues with jobs not correctly cancelling -- I tried to debug with Marvin and determined that this is most likely a bug. Where's the best place to report this / figure out a temporary work around https://prefect-community.slack.com/archives/C04DZJC94DC/p1757441670401089