Sorry, I encountered an error while processing your request. Please try again.

That message usually means the same flow run was submitted twice, and Prefect intentionally rejected the second submission. What it means - A worker/runner tried to move a run to

PENDING

- But the run was already in

PENDING

- Prefect aborts that duplicate transition to avoid double-starting the run So this is often a race condition / duplicate submission protection, not the root failure by itself. Common causes - Two workers polling the same work pool / queue and both grabbing the same run - A submission request timed out client-side, then retried after the server had already accepted it - A runner/execution path proposing

PENDING

again when the run is already in a pending sub-state - Older versions with edge cases around

Submitting

/

InfrastructurePending

Important context There is a known related issue here: - GitHub issue #10263 And there were follow-up fixes for duplicate pending proposals in newer Prefect 3.x code paths: - PR #21646 - PR #21411 What to check first 1. Are multiple workers active for the same pool/queue? - If yes, confirm that’s intentional - Make sure you don’t have an old worker still running somewhere 2. Does the run eventually start anyway? - If yes, this may just be a harmless duplicate submission attempt - If no, then something is causing the run to get stuck after the abort 3. What Prefect version are you on exactly? - Since you previously mentioned upgrading to 3.6.6, if that’s still current, this may be environment-specific rather than the older fixed path - If you’re on an older 3.x release, upgrading may help 4. Are there request timeout / connectivity issues between worker and API? - Intermittent network timeouts can produce “server accepted it, client retried it” behavior If the run is getting stuck Look for nearby worker logs such as: - retries -

ReadTimeout

- repeated

submitting flow run ...

- multiple worker IDs submitting the same flow run - transitions between

Scheduled

,

Pending

,

Submitting

, and

Running

Practical next steps - Restart all workers cleanly so only the intended ones are polling - Verify each worker is attached to the expected work pool - Upgrade to the latest Prefect 3 release if you’re not already there - If you suspect API timeout/retry behavior, check network reliability and any custom timeout settings If you want, send: - your exact Prefect version - worker type (

process

,

docker

,

kubernetes

, etc.) - whether the run eventually executes or stays stuck - 20–30 lines of logs around this message and I can help narrow down whether this is harmless duplicate submission or an actual orchestration problem.

An

Internal Server Error

during

prefect deploy

usually means the server/API failed while processing the deployment request — not that your CLI syntax is wrong. Since you’re on Prefect 3.x, the most common buckets are: Most likely causes 1. Database/backend problem - Prefect server can’t complete a DB transaction - timeouts / connection pool exhaustion / slow Postgres - this is a very common source of 500s 2. Bad deployment config that triggers a server-side exception - invalid

job_variables

- bad block reference - bad work pool name / queue / template mismatch - malformed schedule / parameter schema 3. Work pool or block mismatch - deployment points at a pool, queue, storage, or infra config that no longer exists - or exists but doesn’t match the expected worker type What I’d check first - Does

prefect deploy

fail for all flows or just one? - Are you using Prefect Cloud or a self-hosted server? - What’s your exact Prefect version? - Do server logs show a traceback at the time of deploy? Quick checks 1. Verify the work pool exists:

bash
prefect work-pool ls

2. Inspect the specific pool:

bash
prefect work-pool inspect <pool-name>

3. Try the simplest possible deployment:

bash
prefect deploy ./path/to/flow.py:my_flow --name test-deploy --pool <pool-name>

If the minimal deploy works, the problem is probably in your

prefect.yaml

, schedule,

job_variables

, or block references. Very important If you’re self-hosting, 500s often come from the server database layer. I found related reports where server logs showed

asyncpg

/ SQLAlchemy

TimeoutError

during API requests. So if you self-host, check the API/server logs immediately when deploy fails. Useful command reference

prefect deploy

is the correct 3.x CLI command: - Deployments docs - prefect.yaml docs Known related issue patterns - database timeout / server overload causing 500s - deployment validation failures surfacing as generic internal errors - stale block or infra references GitHub references: - Random 500 Internal Server Error in UI - Server timeouts due to slow database communication If you paste: - the full

prefect deploy

error output - whether you use Cloud or self-hosted - your

prefect.yaml

or deploy command - the matching server traceback I can help pinpoint the exact cause quickly.

That pattern strongly suggests a server-side/backend issue more than a bad global CLI setup. Because: - some deployments succeed - you’re self-hosted - you’re on 3.6.6 the most likely causes are: Most likely root causes 1. A specific deployment config is triggering the 500 - bad

job_variables

- invalid schedule - stale block reference - work pool template mismatch - parameter/default serialization issue 2. Intermittent DB/API pressure - server can handle some deploys, then times out/fails on others - more likely if multiple deploys are being created in a batch 3. A specific flow/project definition issue - one flow’s deployment metadata is malformed - another flow with simpler config deploys fine What to compare between successful and failing deploys Please compare these fields between one that works and one that fails: -

work_pool_name

-

job_variables

- schedules - parameters/defaults - tags - concurrency settings - storage / source settings - whether the failing one uses any blocks or variables the successful one does not What to do next 1. Run the failing deploy by itself If it consistently fails while others succeed, that points to that deployment’s config. 2. Check the server logs at the exact failure time This is the big one. For self-hosted 500s, the traceback in the API/server log is usually the only place the real cause appears. Look for: -

sqlalchemy

-

asyncpg

-

TimeoutError

-

IntegrityError

-

ValidationError

- block/work pool lookup failures 3. Try stripping the failing deployment to minimum config For example:

bash
prefect deploy ./path/to/file.py:flow_name --name test --pool <your-pool>

If that works, add back: - schedule - tags -

job_variables

- parameters one at a time until it breaks. 4. Check server health / DB pressure If this happens during bursts of deploys, inspect server/container logs and DB logs for timeouts. Why I suspect DB/server issues too There are known self-hosted patterns where Prefect returns generic 500s while the actual problem is a database timeout or transaction failure: - Server timeouts due to slow database communication - Random 500 Internal Server Error My best guess from what you’ve shared If it’s not all deployments, then the ranking is probably: 1. specific failing deployment config 2. intermittent DB timeout under load 3. stale referenced object in the failing deployment If you want, paste: - one successful deploy spec/command - one failing deploy spec/command - the server traceback for the failure and I can help diff them and narrow it down fast.

Probably not by itself. A client at

3.6.6

talking to a self-hosted server at

3.7.0

can cause compatibility issues, but a minor-version mismatch alone would not usually explain “some deployments succeed, some fail” unless you’re hitting a feature/schema path that changed between those versions. Short answer - Yes, version skew could contribute - No, it’s not the most likely sole cause Why If the mismatch were fundamentally incompatible, I’d expect more consistent failures: - all deploys failing - obvious API/schema errors - repeatable failures on every request Instead, your pattern suggests either: - only certain deployment payloads hit a changed validation/code path, or - the real issue is still server-side config/DB/validation and the version skew is just making it harder to diagnose Best practice For self-hosted Prefect, keep: - CLI/client version - worker version - server version as closely aligned as possible, ideally exactly the same. So yes — I would upgrade your local/client environment from

3.6.6

to

3.7.0

and retest. Why that helps It removes one variable: - request payload generation matches server expectations - deployment schema handling is aligned - fewer surprises around validation or newer fields Recommended next step Upgrade the environment that runs

prefect deploy

to

3.7.0

, then retry the same failing deployment. If it still fails after versions match, the real cause is almost certainly in: - that deployment’s config - server traceback - DB/API health Also check your workers Since you upgraded from 2.x previously, make sure your workers are also on the intended 3.x version and not mixed across hosts. If you want, send: - the failing section of your

prefect.yaml

- how you installed Prefect - whether your workers are also

3.7.0

and I can help you do a quick compatibility sanity check.

That makes this sound much more like a self-hosted server/database issue than a

prefect.yaml

version mismatch. If the same general deploy process: - works once, - then fails after a few, - and the failure point moves around, that usually points to intermittent backend instability such as: Most likely causes 1. Database timeouts / slow commits 2. API server resource pressure - CPU / memory spikes - container restarts - too few replicas/processes 3. Connection pool exhaustion 4. Reverse proxy / ingress timeout - if deploy requests pass through nginx/traefik/etc. 5. Only secondarily: version skew between

3.6.6

and

3.7.0

Why I think this A bad deployment definition usually fails consistently. A flaky “fails after one, or after a few” pattern is classic: - timeout - overloaded DB - transaction contention - unstable API process There are related issue patterns around Prefect self-hosted 500s caused by DB timeouts: - Server timeouts due to slow database communication - Random 500 Internal Server Error What to check right now Please inspect your Prefect server/API logs at the exact moment of a failed deploy. You’re looking for lines like: -

Encountered exception in request

-

TimeoutError

-

sqlalchemy.exc.*

-

asyncpg.*

-

OperationalError

-

IntegrityError

If you see

TimeoutError

or DB-related stack traces, that’s almost certainly the culprit. High-value checks 1. Server logs - the real reason for a 500 is usually only there 2. Database logs - slow queries - connection exhaustion - restarts - lock waits 3. Server resources - CPU / memory - OOM kills - pod/container restarts 4. Proxy/ingress timeout - if your API is behind nginx, ALB, Traefik, etc. What I’d do next 1. Match versions anyway Upgrade the deploying environment to

3.7.0

so client/server match. This is good hygiene, but I don’t think it’s the main issue. 2. Run one deploy at a time slowly Add a short pause between deploys. If that reduces failures, it further supports backend pressure. 3. Try a minimal deployment repeatedly

bash
prefect deploy ./path/to/file.py:flow_name --name smoke-test --pool <pool>

If even this intermittently 500s, the problem is almost certainly server/DB-side. 4. Check whether the API health endpoint is stable If self-hosted: - does the server stay healthy during repeated deploys? - any restart events? My current best guess Given your symptoms, I’d rank it: 1. DB/API timeout or resource issue 2. server bug triggered under repeated deployment creation 3. version skew 4. specific deployment config issue If you paste the server traceback from one failed deploy, I can probably tell you in one pass whether it’s: - DB timeout - validation bug - work pool/block issue - proxy/network problem

✅ thought for 18.8 seconds

That error is not a Prefect deployment error — it’s coming from Kubernetes / Kopf inside the cluster. What it means Something in your K8s environment is trying to call:

<https://10.231.0.1:443/apis/ext.cattle.io/v1>

and getting:

503 Service Unavailable

<http://ext.cattle.io|ext.cattle.io>

is a Rancher-related API group, so this points to a cluster API / Rancher extension availability problem, not to your flow code itself. Why your flows aren’t running If you’re using a Kubernetes worker, Prefect needs the cluster API to be healthy enough to create/read/watch jobs or pods. If cluster-side API calls are failing with

503

, the worker may not be able to submit or monitor flow-run infrastructure correctly. Most likely causes - Rancher / extension API temporarily unavailable - Kubernetes API server instability - in-cluster DNS/networking issue - RBAC/service account issues surfacing through API access - cluster control plane under load - a controller/operator in the cluster retrying and failing Important detail This specific message is from

kopf

, which is a Kubernetes operator framework. So either: - Prefect is running alongside an operator using

kopf

, or - one of your infra components/controllers is logging this while Prefect waits on K8s resources So the immediate problem seems to have shifted from deployment creation to cluster execution health. What to check 1. Are you using a Kubernetes work pool/worker? If yes, this is very likely the reason runs aren’t starting. 2. Check worker logs around submission Look for: - pod creation errors - job creation errors - auth/RBAC failures - API timeouts -

403

,

503

,

connection refused

3. Check cluster health From a machine with cluster access:

bash
kubectl get pods -A
kubectl get nodes
kubectl get events -A --sort-by=.lastTimestamp
kubectl api-resources | grep cattle

4. Check whether that API group is healthy

bash
kubectl get --raw /apis/ext.cattle.io/v1

If that fails with 503 too, the issue is definitely cluster-side. 5. Check the Prefect worker pod logs Especially the pod running your Kubernetes worker. What this likely is not - not caused directly by

prefect.yaml

- not likely caused by

3.6.6

vs

3.7.0

- not a normal flow-code failure Likely conclusion Your deployment may now exist correctly, but flow execution is blocked by Kubernetes control-plane / Rancher API instability. If you want, send: - whether you’re using a Kubernetes worker - the worker log lines before and after this message - the result of:

bash
kubectl get --raw /apis/ext.cattle.io/v1

and I can help narrow down whether this is: - Rancher API degradation - cluster auth/RBAC - worker pod network issue - a non-Prefect operator failing in the same namespace.