@Marvin Error log is as such ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 244, in run_asgi result = await self.app(self.scope, self.asgi_receive, self.asgi_send) # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call await super().__call__(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 151, in call await self.app(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 77, in call await self.app(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 716, in call await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 736, in app await route.handle(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 462, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call await super().__call__(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 151, in call await self.app(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 19, in call await self.app(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 716, in call await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 736, in app await route.handle(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 364, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 97, in app await wrap_app_handling_exceptions(app, session)(scope, receive, send) File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 95, in app await func(session) File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 384, in app await dependant.call(**solved_result.values) File "/usr/local/lib/python3.12/site-packages/prefect/server/api/events.py", line 72, in stream_events_in event = Event.model_validate_json(event_json) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/pydantic/main.py", line 746, in model_validate_json return cls.__pydantic_validator__.validate_json( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pydantic_core._pydantic_core.ValidationError: 4 validation errors for Event occurred Field required [type=missing, input_value={'type': 'auth', 'token': None}, input_type=dict] For further information visit https://errors.pydantic.dev/2.11/v/missing event Field required [type=missing, input_value={'type': 'auth', 'token': None}, input_type=dict] For further information visit https://errors.pydantic.dev/2.11/v/missing resource Field required [type=missing, input_value={'type': 'auth', 'token': None}, input_type=dict] For further information visit https://errors.pydantic.dev/2.11/v/missing id Field required [type=missing, input_value={'type': 'auth', 'token': None}, input_type=dict] For further information visit https://errors.pydantic.dev/2.11/v/missing

That traceback means the server is trying to parse an auth handshake message as a Prefect Event. The key clue is this payload:

json
{"type": "auth", "token": null}

/api/events/in

expects: 1. a websocket auth handshake first 2. then actual event payloads A real event must include fields like: -

occurred

-

event

-

resource

-

id

Your message has none of those, so Pydantic raises:

text
ValidationError: 4 validation errors for Event

Most likely cause Your client and server are not speaking the websocket protocol the same way. In practice, this usually means one of these: 1. Prefect version mismatch - I found a recent issue where a 3.6.16 worker was talking to a 3.4.25 server and websocket auth failed: Workers cannot connect to workpool 2. A reverse proxy / ingress is interfering with websocket upgrade or subprotocol handling - If the websocket subprotocol/header is stripped, the server can fall back incorrectly and treat the auth message as a normal event payload. 3. A non-Prefect or custom websocket client is sending

{"type":"auth","token":null}

directly to

/api/events/in

- That endpoint is not for arbitrary JSON messages. What to check first 1. Make sure all Prefect components are on the same 3.x version - Especially: - server - workers - clients creating deployments/runs If they differ, upgrade them to the same current release. 2. If you are behind nginx / ingress / load balancer, verify websocket support - Must preserve: -

Upgrade

-

Connection

- websocket traffic to

/api/events/in

- If applicable, also ensure websocket subprotocols are preserved. 3. If server auth is enabled, verify the token is actually set - Check whether

PREFECT_SERVER_API_AUTH_STRING

is configured. - If it is configured, a

null

token will fail auth. - If it is not configured,

null

may be acceptable during handshake, but only if the handshake path is handled correctly. 4. Turn on debug logs on the client/worker - Prefect usually logs websocket auth/connect details more clearly in debug mode:

bash
   export PREFECT_DEBUG_MODE=1
   export PREFECT_LOGGING_LEVEL=DEBUG

Why this happens internally The relevant server handler is

prefect.server.api.events.stream_events_in

, which validates incoming websocket text as an

Event

. If the auth handshake is not consumed properly first, this line fails:

text
event = Event.model_validate_json(event_json)

So the problem is generally not that your event schema is wrong — it’s that an auth message is reaching the event parser. Recommended fix - Upgrade server and workers to the same latest Prefect 3.x version - Check proxy/ingress websocket handling - If using custom code, do not send raw auth payloads to

/api/events/in

unless you’re implementing the full Prefect websocket protocol Helpful references: - Prefect Events - Event specification - Related GitHub issue If you want, send me: - your Prefect server version - worker version - whether you’re using nginx/ALB/ingress - whether

PREFECT_SERVER_API_AUTH_STRING

is set and I can help narrow it down quickly.

✅ thought for 130.3 seconds

That message usually means the worker tried to claim a flow run for execution by moving it to

PENDING

, but the server said “no, this run should stay

SCHEDULED

instead.” So in short: the worker did not fail to submit arbitrarily; orchestration rejected the state transition. Most common reason: concurrency limits The most common cause is that the run hit a deployment/work-pool/work-queue concurrency rule, so Prefect rescheduled it instead of letting it enter

PENDING

. A very typical pattern is: - worker picks up a run - worker proposes

PENDING

- server sees no concurrency slot is available - server returns

SCHEDULED

(often shown in the UI as

AwaitingConcurrencySlot

) - worker logs:

text
  Aborted submission of flow run '...': Server returned a non-pending state 'SCHEDULED'

This is generally expected behavior, not a bug by itself. Related issue: Tasks stuck in AwaitingConcurrencySlot: Server returned a non-pending state 'SCHEDULED' Other possible causes - Another worker already grabbed the same run - Prefect prevents duplicate submission, so a second attempt can get rejected. - The run was rescheduled by orchestration logic - For example, because it should not start yet. - A pause/cancel/orchestration rule intervened - Less common, but same general mechanism. What this means operationally If the run is being rescheduled due to concurrency, the worker is behaving correctly. The run should wait and be retried later. If it stays stuck in

SCHEDULED

/

AwaitingConcurrencySlot

for too long, then check: - deployment concurrency limits - work pool / work queue concurrency - whether another run is holding the slot - whether a prior run is stuck in

RUNNING

or otherwise not releasing concurrency What to inspect - In the UI, check the run’s current state name - If it says

AwaitingConcurrencySlot

, that strongly points to concurrency enforcement. - Check your deployment settings for concurrency limits. - Check whether multiple workers are polling the same queue and competing for the same run. - Check whether an earlier run is stuck and blocking the slot. Why the wording is confusing The word

Aborted

here comes from the worker’s submission step, not necessarily from a system error. It means: - “I tried to submit this run now” - “the API responded with a different state” - “so I won’t launch infrastructure for it” It does not necessarily mean the flow run is broken. Relevant docs: - Work pools - Deployments - Flows If you want, I can help pinpoint it if you share: - whether the run state in UI says

AwaitingConcurrencySlot

- your deployment concurrency limit - your work pool / queue concurrency settings - whether you have multiple workers polling the same pool