<@ULVA73B9P> did something change in the last day ...
# ask-marvin
j
@Marvin did something change in the last day where prefect worker start --work-queue is required ?
m
thought for 99.9 seconds
No — in current Prefect 3.x,
--work-queue
is not required. I verified the CLI help for
prefect worker start
, and only
--pool
is required.
--work-queue
is optional and defaults to polling all queues in the work pool. So this is valid:
Copy code
bash
prefect worker start --pool my-pool
And
--work-queue
is only needed if you want to restrict the worker to specific queues:
Copy code
bash
prefect worker start --pool my-pool --work-queue queue-a --work-queue queue-b
I also checked for recent changes and didn’t find anything indicating that this became required in the last day. Relevant docs: - Workers documentation If you’re seeing an error that implies
--work-queue
is required, paste the exact command and error text here and I can help pinpoint what’s actually going on.
j
@Marvin I am starting my work pool without any work queues specified, I have two queues. The work pool is in status ready, but the queues are in status not ready The dashboard on the queue says that it's "ready to go" in the title bar but status in the right sidebar is Not Ready Why would my worker be ready and my queue be not ready?
m
thought for 122.9 seconds
Yes — this can happen, and it’s because work pool readiness and work queue readiness are tracked differently. Short version: - Work pool = Ready means Prefect sees an online worker for that pool. - Work queue = Not Ready means that queue has not been polled recently. So a worker can make the pool look healthy while the queues still show Not Ready if Prefect hasn’t recorded recent polling for those queues. What Prefect uses for each status: - Work pool readiness is based on worker presence / heartbeats - Work queue readiness is based on whether it has been polled within roughly the last 60 seconds If a worker starts with just:
Copy code
bash
prefect worker start --pool my-pool
then it should poll all queues in that pool, and those queues should typically become
Ready
after polling. So if the pool is
Ready
but the queues are
Not Ready
, likely explanations are: 1. The worker is online, but not actually polling those queues successfully - heartbeat is reaching the API - queue polling may be failing or not happening 2. A transient UI/backend timing issue - statuses are updated asynchronously - you may briefly see inconsistent state between pool and queues 3. The worker is connected to the pool, but filtering/config means it is not polling the queues you expect - less likely if you truly started it with only
--pool
4. The queues aged out of readiness - queue readiness is time-based - if a queue hasn’t been polled recently, it flips back to
Not Ready
About the conflicting UI text: - The queue page title/header saying “ready to go” sounds like more of a generic UI message - The sidebar status is more likely reflecting the actual stored queue readiness state So if those disagree, I’d trust the explicit
Status: Not Ready
more than the page title copy. What I’d check next: 1. Worker logs Look for repeated polling activity or errors. You want to see signs it is: - connected to the correct pool - requesting scheduled flow runs - not erroring on API calls 2. Pool/queue names Confirm the worker is attached to the exact pool you’re viewing in the UI. 3. Whether runs are actually being submitted to those queues If deployments target a different pool or queue, the worker may be healthy but irrelevant to those queues. 4. Refresh after ~1 minute Since queue readiness is tied to recent polling, it should usually settle quickly if polling is working. If you want, paste: - your worker start command - a snippet of worker logs from startup and a few polling cycles - whether you’re on Cloud or Server / self-hosted and I can help narrow down why the queues are staying
Not Ready
.
j
@Marvin How much time elapses before the queues age out of readiness?
m
thought for 4.2 seconds
About 60 seconds. In Prefect 3.x, a work queue is considered
READY
if it has been polled by a worker within the last 60 seconds. If it goes longer than that without a poll, it will age back to
NOT_READY
. Docs: - Work pools and work queues So practically: - polled within ~1 minute →
READY
- no poll for > ~1 minute →
NOT_READY
If you want, I can also help you map that to the worker polling cadence you should expect in logs.
j
@Marvin Is that poll time configurable?
m
thought for 105.6 seconds
Yes — the queue readiness timeout is configurable. By default it’s 60 seconds, but on Prefect Server/self-hosted you can change it with:
Copy code
bash
PREFECT_SERVER_SERVICES_FOREMAN_WORK_QUEUE_LAST_POLLED_TIMEOUT_SECONDS
There’s also a legacy alias:
Copy code
bash
PREFECT_API_SERVICES_FOREMAN_WORK_QUEUE_LAST_POLLED_TIMEOUT_SECONDS
And the worker polling interval is separately configurable with:
Copy code
bash
PREFECT_WORKER_QUERY_SECONDS
Defaults are roughly: -
PREFECT_WORKER_QUERY_SECONDS=10
-
PREFECT_SERVER_SERVICES_FOREMAN_WORK_QUEUE_LAST_POLLED_TIMEOUT_SECONDS=60
So the relationship is: - worker polls every ~10s by default - queue is marked
Not Ready
if it hasn’t been polled for ~60s If you’re self-hosting, you can tune both. If you’re using Prefect Cloud, server-side foreman settings generally aren’t user-configurable. Relevant docs: - Work pools and work queues - Settings reference If you want, I can suggest sane values depending on whether your workers are long-lived, bursty, or running in Kubernetes.
j
@Marvin What does this DEBUG log message in my work pool logs mean? Worker channel unhealthy, REST fallback is active: received 1013 (try again later) service_unavailable; then sent 1013 (try again later) service_unavailable
m
thought for 68.8 seconds
That message means the worker’s preferred realtime connection to the Prefect API became unhealthy, so it temporarily fell back to plain REST API communication. In practice: - Prefect workers try to maintain a WebSocket-based worker channel - that channel is used for more realtime coordination with the API - if the channel becomes unavailable, the worker does not just stop - instead, it switches to a REST fallback path and keeps working while it retries the channel The specific part:
Copy code
text
received 1013 (try again later) service_unavailable
means the WebSocket was closed with code
1013
, which generally means: - service unavailable - temporary overload - try again later So this usually points to a temporary API-side or network-side disruption, not necessarily a fatal worker problem. What this log implies: - good news: the worker detected the channel problem and handled it correctly - good news: fallback is active, so the worker can often continue operating - possible concern: the API, proxy, load balancer, or network path briefly refused or interrupted the WebSocket connection When to worry: - If you see it occasionally: usually not a big deal - If you see it constantly or continuously: then something is persistently breaking WebSocket connectivity Common causes: - temporary Prefect API unavailability - reverse proxy / ingress timeouts on WebSockets - load balancer behavior - transient network instability - server under load If your flows are still being picked up and executed, this log by itself is usually just informational/debug noise indicating degraded-but-functional behavior. If runs are not being picked up, then I’d investigate: 1. worker logs around the same timestamp 2. API / server health 3. any ingress or proxy in front of Prefect 4. whether the message repeats every few seconds/minutes Docs: - Workers If you want, paste a larger log snippet around that message and I can help tell whether it’s harmless fallback behavior or a real connectivity problem.
j
@Marvin Could a corporate firewall prevent my self-hosted worker from connecting? What would I have IT look for in their logs?
m
thought for 17.1 seconds
Yes — absolutely. A corporate firewall, proxy, or SSL-inspecting gateway could prevent a self-hosted worker from establishing the WebSocket worker channel, even if normal HTTPS requests still work. That would fit what you saw: - worker can sometimes reach the API - WebSocket gets closed or rejected - Prefect falls back to REST - logs show
1013 ... service_unavailable
What to ask IT to check Tell them your worker needs outbound connectivity from the worker host/container to your Prefect API endpoint over HTTPS / WSS. Specifically, ask them to check for: - blocked outbound WebSocket connections to the Prefect API hostname - TLS inspection / SSL interception that may interfere with upgraded WebSocket connections - proxy/load balancer timeouts on long-lived connections - HTTP Upgrade header stripping or rejection - idle connection termination - rate limiting / service unavailable responses coming from a proxy or gateway - destination hostname or port filtering for the Prefect API URL Useful wording for IT You can give them something like:
Copy code
text
We have a service that needs outbound HTTPS and secure WebSocket (WSS) connectivity to <prefect-api-hostname>:443.

Please check whether any firewall, proxy, SSL inspection appliance, or ingress layer is:
- blocking or resetting WebSocket connections
- rejecting HTTP Upgrade requests
- closing long-lived idle connections
- returning 1013 / service unavailable responses
- performing TLS interception that breaks WSS
What should work Your worker should be able to connect to: -
https://...
for normal REST API traffic -
wss://...
for the worker channel If REST works but WSS does not, that strongly suggests: - firewall/proxy rules - ingress config - or SSL interception issues What evidence IT should look for in logs Ask them to search logs for the worker machine’s source IP and the Prefect API destination, especially around the timestamps where you see the debug message. Helpful indicators: - denied or dropped outbound
CONNECT
or
Upgrade: websocket
requests - proxy responses like
403
,
502
,
503
,
504
- connection resets / TCP resets - TLS handshake failures - certificate validation or certificate substitution events - WebSocket upgrade failures - long-lived connection timeout or idle timeout events - service-unavailable responses generated by a proxy/load balancer rather than the Prefect app itself Headers / protocol details they may care about If they inspect HTTP logs, they should look for WebSocket upgrade attempts like: -
Connection: Upgrade
-
Upgrade: websocket
- requests over HTTPS to your Prefect API host - successful
101 Switching Protocols
versus failure responses If they never see a successful WebSocket upgrade, that’s a clue. A few practical checks you can do yourself From the worker environment, test: 1. basic HTTPS reachability to the API 2. whether a proxy is configured via
HTTP_PROXY
,
HTTPS_PROXY
,
NO_PROXY
3. whether the worker sits behind: - corporate proxy - Kubernetes egress proxy - ingress/load balancer with short idle timeout Important nuance If you are self-hosting Prefect Server, the problem may also be on your own ingress/reverse proxy in front of the API, not just the corporate firewall. In that case, check: - nginx / envoy / ALB / ingress controller WebSocket support - idle timeout values - upgrade header forwarding - 503 generation upstream If you want, I can help you build a short network checklist for IT based on whether your worker is: - on a VM - in Docker - in Kubernetes - behind a corporate proxy - connecting to Prefect Cloud vs self-hosted Prefect Server
j
@Marvin I am self-hosting the workers, but I am connecting to Prefect cloud for the prefect server.
m
thought for 14.2 seconds
Yes — in that setup, a corporate firewall/proxy is a very plausible cause. Since your workers are inside your network but connecting outbound to Prefect Cloud, the important distinction is: - REST HTTPS traffic to Prefect Cloud may still work - but WebSocket/WSS traffic used by the worker channel may be blocked, intercepted, downgraded, or timed out That would produce exactly the sort of behavior you described: - worker appears online / pool may look ready - queue readiness may be inconsistent - debug logs mention worker channel unhealthy and REST fallback is active What to ask IT to verify Ask them to verify that the worker hosts have outbound access to Prefect Cloud over: - HTTPS (443) - WSS over 443 to the same Prefect Cloud endpoint path/domain used by your API client They should specifically check for devices or policies that: - block WebSocket upgrades - terminate long-lived outbound TLS sessions - do SSL/TLS interception - require an authenticated proxy the worker is not using - enforce short idle timeouts - return or inject
503
/
service unavailable
- rate-limit or close upgraded connections Good message to send IT
Copy code
text
We run Prefect workers inside our network that make outbound connections to Prefect Cloud.

The workers need stable outbound HTTPS and secure WebSocket (WSS) connectivity over port 443 to Prefect Cloud.

Please check whether any firewall, proxy, SSL inspection appliance, or egress control is:
- blocking outbound WSS / WebSocket connections
- rejecting HTTP Upgrade requests
- intercepting TLS and replacing certificates
- closing long-lived connections due to idle timeout
- returning 503/service unavailable responses
- resetting connections after upgrade
What they should look for in logs Have them search egress logs around the exact timestamps of your worker debug messages for the worker node/IP and Prefect Cloud destination. Useful things to look for: - outbound denies to Prefect Cloud hostname on port 443 - proxy errors on
CONNECT
- WebSocket upgrade failures -
403
,
429
,
502
,
503
,
504
- TCP reset / connection terminated events - TLS inspection/certificate substitution events - idle session timeout events - policy engine events for “unknown application”, “websocket”, “tunneling”, or “upgrade” Important nuance for Prefect Cloud Because this is outbound to a SaaS service, the most likely culprits are: - corporate web proxy - SSL inspection appliance - egress firewall - CASB / secure web gateway This is less likely to be an ingress/reverse-proxy misconfiguration on your side, unless your workers must traverse an internal outbound proxy first. What you can tell if REST fallback is working If the worker still: - heartbeats - stays visible in Cloud - sometimes picks up runs then basic HTTPS is probably okay. If the WebSocket worker channel keeps failing, that usually means: - WSS specifically is degraded or blocked - or long-lived connections are being interrupted What I’d ask you to check locally too From the worker environment: - whether
HTTPS_PROXY
,
HTTP_PROXY
, or
NO_PROXY
are set - whether all outbound traffic must go through a corporate proxy - whether certificate trust is customized - whether disconnects happen on a regular cadence, like every 30s / 60s / 5m, which often points to enforced timeout policy Prefect Cloud links: - Prefect Cloud - Workers If you want, I can help you turn this into a very short IT ticket template plus a self-check command list for your worker host/container.