It sounds like a timezone mismatch between what the UI previews and how the scheduler is actually triggering. Key points for Prefect 3.x: - Timezone is set per schedule (not per deployment). If you don’t set a timezone on a Cron schedule, it defaults to UTC. - The UI “Schedules” sidebar previews times in your browser’s local timezone, but the scheduler will trigger runs in the schedule’s timezone (UTC if not specified). - Result: if your cron has no timezone, you may see 03:30 AM runs when you expected 03:30 PM locally. How to fix - Explicitly set the timezone on the schedule in your deployment (use an IANA tz like America/New_York or Europe/London). YAML example (single schedule for all three times):

deployments:
  - name: my-deployment
    entrypoint: path/to/flow.py:my_flow
    schedules:
      - cron: "30 7,11,15 * * *"
        timezone: "Your/Timezone"   # e.g., "America/Chicago"

Or CLI:

# See your current schedules (and confirm timezone)
prefect deployment schedule ls "flow-name/deployment-name"

# Clear existing schedules (optional, if you want to replace them)
prefect deployment schedule clear "flow-name/deployment-name" -y

# Recreate with an explicit timezone
prefect deployment schedule create "flow-name/deployment-name" \
  --cron "30 7,11,15 * * *" \
  --timezone "Your/Timezone"

What to check right now - In the UI: Deployment > Schedules > click your schedule and confirm the timezone shown. - In the CLI:

prefect deployment schedule ls "flow-name/deployment-name"

and look for the timezone field. - If it’s missing or “UTC”, that explains the 03:30 AM run instead of 03:30 PM local. DST notes - With a timezone set, cron schedules follow local clock time across DST shifts (e.g., “15:30” stays 3:30 PM local time). - Details: Create schedules and Schedule concepts. If your schedule already has the correct timezone set and you’re still seeing 03:30 AM runs, let me know: - Your intended timezone - What the UI shows for the schedule’s timezone - The deployment name and the

next_scheduled_start_time

for a recent run (as displayed in the Flow Runs table) I’ll help narrow it down from there.

Got it — if the only concurrency controls you have are per-deployment limits of

1

, then Prefect itself is not imposing any broader org/work-pool/tag/global concurrency limit beyond that. That means a few things: - Each individual deployment can only have one active run at a time - Different deployments are still free to run concurrently - If you're seeing less parallelism than expected, the bottleneck is likely not an additional Prefect concurrency limit The next places I’d check are: 1. Worker capacity / infrastructure capacity - Do you have enough workers or execution slots? - If using Kubernetes/ECS/Docker/etc., is the underlying infrastructure actually able to launch multiple flow runs? 2. Work pool / worker type behavior - Some worker setups effectively serialize work if only one worker process is polling and submitting slowly, or if infra is constrained. 3. Task runner inside the flow - Even if multiple flow runs can happen, tasks within a flow may still execute sequentially unless you’re using concurrency explicitly. 4. Automation / scheduling pattern - If many runs are being created but staying in

Late

,

Scheduled

, or

Pending

, that usually points to worker or infrastructure pickup issues rather than Prefect concurrency limits. 5. Code-level locking / external resource contention - Database locks, file locks, shared temp paths, or application-level mutual exclusion can make it look like Prefect is throttling. If you want, I can help narrow it down quickly — just share: - your worker type (

process

,

docker

,

kubernetes

, etc.) - whether the issue is - multiple deployments not running at the same time, or - tasks inside one flow not running in parallel - what state the blocked runs sit in (

Scheduled

,

Pending

,

Running

,

Late

, etc.) And if helpful, here are the relevant docs: - Global concurrency limits - Work pools - Deployments

I wouldn't recommend setting one broad global concurrency limit of

20

and treating it as a universal safety valve unless all those processes compete for the same bottleneck. A better mental model is: - Deployment concurrency limit = "how many runs of this specific workflow can overlap?" - Global concurrency limit = "how many things can touch this shared resource at once?" - Tag-based concurrency = "how many tasks of this class can run across flows?" So in your case, since workloads are heterogeneous and some are much heavier than others, a single global limit like

20

can still let you get unlucky: - 20 light runs might be fine - 20 heavy runs might overwhelm CPU / memory / DB / IO - a bad mix of several heavy runs colliding can still hang things So yes, there is still a chance multiple heavy processes coinciding would hang again. My recommendation: 1. Keep deployment limits where collisions are known to be unsafe - If a deployment must never overlap with itself, keep it at

1

2. Do not start with one large catch-all global limit unless you know the shared bottleneck - If the issue is actually DB connections, rate-limited APIs, GPU slots, disk pressure, etc., limit that resource specifically 3. Group workloads by heaviness - Heavy deployments: low concurrency, maybe

1-2

- Medium deployments: moderate concurrency - Light deployments: higher concurrency 4. Ramp conservatively - Start smaller than

20

- Increase gradually while watching: - worker utilization - host CPU / memory - DB connection usage - queueing / late runs - failure rate / hangs A practical approach could be: - leave self-colliding deployments at

1

- identify "heavy" deployments and cap them individually - optionally add a shared limit for heavy work as a class For example, if you know only 3 heavy jobs can safely run at once across the whole system, use a shared mechanism for that rather than a blanket

20

. If you're doing this at the task/resource level, Prefect global concurrency limits are useful: Global concurrency limits If you're coordinating tasks across flows by category, tag-based limits may be a better fit: Tag-based concurrency limits And deployment-specific limits are here: Deployment concurrency limiting Rule of thumb: - Use deployment limits to prevent same-flow overlap - Use global/tag-based limits to protect shared resources or workload classes - Don’t use a single global number as a proxy for actual system capacity unless your workloads are pretty uniform If you want, I can help you sketch a concrete concurrency strategy from your current setup — e.g.: -

heavy

jobs capped at

2-3

-

medium

jobs capped at

5

-

light

jobs mostly unrestricted - plus a shared DB/API limit if needed If you share what kind of "hang" you were seeing (CPU pinned, OOM, DB lock, too many subprocesses, etc.), I can suggest a more precise limit structure.

Yes — what you’re seeing is consistent with a concurrency slot still being held by runs that are in

Cancelling

, so new runs are now getting

AwaitingConcurrencySlot

. A few important points: 1.

Cancelling

is not the same as gone A flow run in

Cancelling

often still has live infrastructure/processes behind it. Prefect is trying to shut it down cleanly. Until that fully resolves, the concurrency slot may remain occupied. 2. If your worker is on-prem, you can absolutely stop the underlying process yourself Prefect does not magically own the OS process lifecycle beyond what the worker can reach and terminate. If you know the worker launched local processes and they’re hanging, the practical path is: - identify the running OS processes on the worker host - kill them directly on that machine - then let Prefect catch up / mark them cancelled, or wait for lease/slot cleanup If these are process worker jobs running on the same host, this usually means using normal host-level tooling like: -

ps

-

top

/

htop

-

kill

-

kill -9

as a last resort If you stop the actual underlying process, that’s the fastest way to stop consuming resources. 3. But killing the process and releasing the Prefect slot are related, not identical Even after you kill the OS process, Prefect may take a little time to observe the infrastructure is gone and release the slot. So the operational order is usually: - kill the stuck local processes on the worker machine - make sure the worker is still running and can report state back - if needed, wait a short bit for Prefect to transition those runs out of

Cancelling

If the worker itself is unhealthy, restarting the worker after the orphaned processes are gone can help it reconcile state. 4.

AwaitingConcurrencySlot

means some concurrency mechanism is definitely active Since you said this state is new, one of these is almost certainly happening now: - a deployment concurrency slot is still held by one of those cancelling runs - a global concurrency limit was added - a tag-based concurrency limit exists on one or more tasks Given your earlier note, the most likely cause is the deployment-level concurrency limit of 1 still being occupied by a run that hasn’t fully exited. 5. If your goal is “I need this thing dead now”, the most direct action is on the worker host, not in Prefect UI Since you’re on-prem, Prefect can only do so much. If a process is wedged, the fastest practical answer is host-level process management. About safe execution limits for mixed workloads You’re also getting at a real limitation: Prefect cannot automatically infer “safe” capacity for heterogeneous jobs. And no — Prefect cannot guarantee “if scheduled for 10:00, this flow will be done by 10:15” just from scheduling/concurrency settings. Prefect can help with: - not starting before a scheduled time - limiting overlap - queuing when capacity is exhausted But it cannot guarantee: - runtime duration - no host contention - no heavy-job collision unless you model that explicitly So your instinct is right: if developers “shouldn’t need to know about all other runs,” then you need platform-level workload classes, not ad hoc tribal knowledge. A good pattern for your case would be: - keep per-deployment concurrency at

1

where self-collision is unsafe - create a small number of workload classes like: -

heavy

-

medium

-

light

- apply shared limits to those classes, instead of making each dev reason about every other deployment That way developers only choose: - “this is heavy” - “this is medium” - “this is light” and the platform enforces the rest. Since you said the jobs are mostly: - I/O bound - then CPU bound - then memory bound that suggests your first shared safety control should probably be around CPU-heavy classes or host saturation, not just raw flow count. Practical next steps right now

1. On the on-prem worker machine, find and kill the stuck local processes 2. Confirm whether the worker is still healthy 3. Check whether the stuck runs are all from the same deployment with concurrency

1

4. Once the immediate fire is out, define 2–3 workload classes with explicit concurrency ceilings Important caveat If you forcibly kill worker-host processes, you may leave runs briefly inconsistent in the UI until Prefect reconciles them. That’s normal in an emergency cleanup situation. Relevant docs: - Cancel workflows - Deployment concurrency limiting - Global concurrency limits - Tag-based concurrency limits - Workers If you want, I can help you with a concrete rescue procedure for your setup, for example: - how to identify orphaned local Prefect processes on the worker host - how to tell whether it’s deployment vs tag vs global concurrency causing

AwaitingConcurrencySlot

- how to design a simple

heavy/medium/light

limit model that your devs can use without thinking about the whole system

Short answer: no — you generally cannot restart an on-prem worker from the Prefect UI. The UI can show worker status and let you manage orchestration objects, but an on-prem worker is just a process running on your infrastructure. Restarting it has to be done through whatever is managing that process on your side, for example: -

systemd

- Docker - Kubernetes - a VM process supervisor - a terminal session / service wrapper Is restarting the worker safe/reliable? - Usually yes as an operational step - but it is not the same as killing stuck flow processes - and it may not immediately fix runs stuck in

Cancelling

if the underlying job processes are still alive So I’d think of it this way: - Restart worker = refreshes polling / orchestration connectivity - Kill underlying job processes = actually stops the bad work consuming resources If the worker itself is wedged, restarting it is reasonable. If the run infrastructure/processes are wedged, restarting the worker alone may not be enough. Best practice in your situation 1. Check for and kill any obviously stuck local run processes on the worker host 2. Then restart the worker service/process 3. Watch whether the

Cancelling

runs move to terminal states and whether

AwaitingConcurrencySlot

clears If your worker is managed as a service, use your platform-native restart method rather than trying to do this through Prefect. For reference: - Workers - Cancel workflows If you tell me how the on-prem worker is running — e.g. systemd, Docker, Kubernetes, or just a shell process — I can give you the safest restart procedure.

That actually helps narrow it down a lot. If you’re using a Docker worker, and

podman stats --no-stream

only shows the worker container and not per-flow containers, then it’s very possible the stuck runs are not hanging on live infrastructure at all. In that case: - the flow run may be stuck in

Cancelling

at the orchestration level - the deployment concurrency slot may still look occupied - new runs then end up in

AwaitingConcurrencySlot

So yes — this can happen even when no actual job container exists. What this usually means is one of these: 1. The run acquired/attempted to acquire a deployment concurrency slot but never got far enough to launch infrastructure 2. The worker lost track of the run during submission/cancellation 3. State reconciliation didn’t complete, so Prefect still thinks the run is mid-cancel In your case, if there are no actual job containers, restarting the worker is much more reasonable and lower-risk than if live infra were still running. What I’d do next: 1. Restart the Docker worker - since there doesn’t appear to be real job infra to kill, this is a sensible first step 2. After restart, watch the stuck runs - do they move from

Cancelling

to

Cancelled

? - does

AwaitingConcurrencySlot

clear? 3. If they stay stuck, inspect which deployment they belong to - if they’re all from the same deployment with concurrency

1

, that strongly suggests the slot is just logically stuck 4. Check whether the worker logs show failed container creation - especially around: - Docker/Podman API errors - cancellation handling - infra submission failures - lost connection to Prefect API The detail that infra may never have been created is important, because that means this is less about “kill bad processes” and more about “Prefect thinks a run is still occupying lifecycle/concurrency state.” Is restarting safe here? Yes, relatively speaking, this is one of the safer times to restart the worker: - no visible child job containers - no obvious active infra to orphan - issue appears orchestration-related rather than resource-cleanup-related It still may not fully resolve the stuck state, but it’s a good next step. One important nuance With deployment concurrency limits, a run can block others even if it never did useful work, if Prefect believes it still owns the slot. So “no container exists” does not necessarily mean “no concurrency slot is held.” If the restart doesn’t clear it, then we’re likely looking at one of: - stale deployment concurrency state - lease cleanup delay - worker/container creation failure that left the run in an awkward transitional state Docs: - Workers - Deployment concurrency limiting - Cancel workflows If you want, I can help you debug this in a very targeted way: - determine whether the worker is using Docker against a Podman socket - what worker logs to look for - how to tell whether the run is stuck before infra creation vs after submission - how to identify whether the deployment concurrency slot is the thing that’s stuck

That sounds like the key issue. If you’re seeing log/UI messages about hitting deployment concurrency limits, then the blockage is very likely deployment-level, not dead Docker/Podman infrastructure. And just to clarify the model: Deployment concurrency limits do not only limit “other deployments.” They limit runs of that same deployment. So if a deployment has concurrency

1

, then: - one run of Deployment A can be active - any additional runs of Deployment A will wait in

AwaitingConcurrencySlot

- Deployment B is unaffected by A’s deployment limit So “they are only supposed to limit each other” is probably the misunderstanding. Deployment concurrency is per deployment, not a shared pool across deployments. That means if you have: - Deployment A limit =

1

- one run of A stuck in

Cancelling

then: - the next run of A will wait in

AwaitingConcurrencySlot

- even if no infra was ever successfully launched That matches what you’re describing exactly. What this implies operationally You likely have a run that: - belongs to a deployment with concurrency

1

- entered

Cancelling

- is still considered to hold that deployment slot - therefore blocks later runs from the same deployment What I’d do next 1. Confirm affected runs are all from the same deployment - if yes, this is almost certainly just the deployment slot being held by the stuck run 2. Restart the worker - since you’re not seeing actual job containers, this is a reasonable reconciliation step 3. Wait a few minutes after restart - the stuck run may reconcile to

Cancelled

- the slot may be released - queued runs may proceed 4. If it still does not clear, temporarily remove or raise that deployment’s concurrency limit - as an emergency unblock, if safe for that workflow This last part matters: if the deployment limit exists only to prevent collisions, and there’s currently no actual live run, then temporarily relaxing it may be the simplest operational workaround. But be careful: only do that if overlapping runs of that deployment are actually safe enough right now. Why this can happen without real infra A run can hold the deployment concurrency slot during orchestration/state transitions even if container creation failed or never happened. So: - no child container - but slot still occupied is possible. Important distinction - Deployment limit = per deployment, blocks more runs of the same deployment - Global concurrency limit = shared resource limit across different flows/deployments - Tag-based limit = shared task-class limit across flows Docs: - Deployment concurrency limiting - Global concurrency limits - Tag-based concurrency limits If you want, I can help you think through the safest immediate unblock: - restart worker and wait - temporarily increase/remove the deployment limit - or restructure the limits so self-collision protection doesn’t become a deadlock during cancellation

Yes — your reading is basically right. What those logs mean is: - the worker picked up the run - the worker attempted to move it into submission / pending execution - the server rejected that attempt because the deployment concurrency limit was already full - so the worker did not actually start infrastructure That’s why you see: - “submitting flow run …” - then immediately - “Aborted submission … Deployment concurrency limit reached” and nothing actually runs. Important nuance about state display In Prefect, the user-visible behavior here can be confusing because the orchestration/state model distinguishes between: - the state type - the state name The “waiting for a deployment slot” behavior is associated with

AwaitingConcurrencySlot

, but depending on where you’re looking, you may still mainly perceive it as a run that is just not progressing out of

Pending

/ scheduled retry behavior. The key operational fact is: the worker is being told not to launch infra because the deployment slot is occupied. So your core conclusion is right: - the run gets picked up - server says “don’t submit, concurrency limit reached” - worker backs off - no useful work starts - this can repeat indefinitely if the deployment slot never clears That matches your logs exactly. Why this can deadlock operationally If a deployment has concurrency

1

, and Prefect believes one run of that deployment still occupies the slot, then every later run of that same deployment will keep getting rejected before infra launch. So you can end up in this situation: - no active useful containers - worker keeps polling - worker keeps trying to submit - server keeps saying “deployment concurrency limit reached” - run never starts - queue appears stuck forever That is not because Podman is the bottleneck. It’s because the orchestration slot is logically occupied. So what should you do right now? Given your description, the immediate issue is not “how do I kill dead containers?” It’s “how do I clear the stuck deployment concurrency holder?” Most likely options: 1. Find the run from that same deployment that is still holding the slot - likely in

Cancelling

,

Running

, or another non-terminal state 2. Restart the worker - since no actual flow containers appear to exist, this is a reasonable reconciliation step 3. Wait a short interval after restart - Prefect may reconcile the stuck run and release the slot 4. If still stuck, temporarily remove/increase the deployment concurrency limit - only if overlapping runs of that deployment are safe enough as an emergency measure Because from what you’ve shown, the worker is not the thing choosing to hold back — the server is explicitly aborting submission due to deployment concurrency. This also answers your earlier concern You asked whether there is a way to safely limit execution without requiring every dev to know the whole system. For self-collision prevention, deployment concurrency is the right primitive. But the downside is exactly what you’re seeing: - if the slot holder gets stuck in orchestration state, - the deployment can effectively deadlock until that slot is released. So deployment concurrency is good for: - “never overlap runs of this same deployment” But it is not a complete solution for: - “make sure the system keeps making forward progress under failure” For that, you need some operational strategy around: - cancellation cleanup - retry / timeout handling - workload-class limits - sometimes temporary manual override My strongest read of your situation Based on your logs: - the worker is healthy enough to poll and attempt submissions - Podman is probably not the immediate problem - the blocking condition is deployment concurrency enforcement on the Prefect side - the root cause is likely a stuck prior run still holding the slot Docs:

- Deployment concurrency limiting - States - Cancel workflows - Workers If you want, I can help you with the next exact debugging step: - how to identify which run is holding the deployment slot - how to tell whether it’s a stuck cancelling run vs lease cleanup issue - and the safest emergency workaround without opening the floodgates for all deployments