<@ULVA73B9P> - Using self hosted prefect 3.6.20. D...
# ask-marvin
c
@Marvin - Using self hosted prefect 3.6.20. Do we need to use PREFECT_SERVER_DOCKET_URL for only background services or also for API Server configuration
m
thought for 150.2 seconds
Short answer: Only background services. The API server does not use PREFECT_SERVER_DOCKET_URL. Details: - PREFECT_SERVER_DOCKET_URL configures Docket (the coordination layer used by background services like the scheduler and automations). - It’s required when you run background services in a multi-server/HA setup (so they coordinate via Redis). In single-node “all-in-one” it defaults to memory:// and you can ignore it. - The API server itself does not read or require PREFECT_SERVER_DOCKET_URL. For the API server, you typically configure: - PREFECT_SERVER_API_HOST / PREFECT_SERVER_API_PORT (bind address) - PREFECT_API_DATABASE_CONNECTION_URL (database) - PREFECT_UI_API_URL (what the browser-based UI should call) - Clients/workers use PREFECT_API_URL to reach your API Examples API server (no Docket needed):
Copy code
export PREFECT_API_DATABASE_CONNECTION_URL="<postgresql+asyncpg://user:pass@db/prefect>"
export PREFECT_SERVER_API_HOST="0.0.0.0"
export PREFECT_SERVER_API_PORT="4200"
export PREFECT_UI_API_URL="<https://prefect.example.com/api>"
# no PREFECT_SERVER_DOCKET_URL needed here
Background services (use Docket; Redis strongly recommended for multi-server):
Copy code
export PREFECT_API_DATABASE_CONNECTION_URL="<postgresql+asyncpg://user:pass@db/prefect>"
export PREFECT_SERVER_DOCKET_URL="<redis://redis:6379/1>"
References - Docket settings in source: ServerDocketSettings (defines PREFECT_SERVER_DOCKET_URL) - UI API URL setting: Server UI settings (PREFECT_UI_API_URL) - API settings (client URL and server binding): API settings - Self-hosting architecture (multi-server vs single-server): Self-hosted guide If you can share whether you’re running an all-in-one node or separate API + services, I can suggest the exact env set you need.
c
API server and background services are running separately and integrated with Redis
@Marvin - API server and background services are running separately and integrated with Redis
m
thought for 80.3 seconds
Great — in your split deployment (API server and background services separated, with Redis): - Set PREFECT_SERVER_DOCKET_URL only on the background services. It’s not used by the API server and can be omitted there. - Docket is the coordination layer for schedulers/automations; with multiple service processes or HA, it must point to Redis. Recommended env splits API server (no Docket needed):
Copy code
# Database
PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@db/prefect>

# Bind and UI
PREFECT_SERVER_API_HOST=0.0.0.0
PREFECT_SERVER_API_PORT=4200
PREFECT_UI_API_URL=<https://your.public.prefect/api>

# Do NOT set PREFECT_SERVER_DOCKET_URL here
Background services (use Docket via Redis):
Copy code
# Same database as API
PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@db/prefect>

# Docket (coordination)
PREFECT_SERVER_DOCKET_URL=<redis://redis-host:6379/1>
# or: rediss://... for TLS, include auth if needed
Notes - Docket is distinct from any Redis you may use for messaging/events; those have separate env vars. Using Redis for messaging does not automatically configure Docket — you still set PREFECT_SERVER_DOCKET_URL for the services. - It’s fine (and common) to use Redis-backed Docket even if you currently run a single services process; it keeps you ready to scale. CLI references - Server (API) start (no Docket option): run
prefect server start --help
- Background services:
prefect server services start --help
Source links - ServerDocketSettings (PREFECT_SERVER_DOCKET_URL) - Server UI settings (PREFECT_UI_API_URL) - Self-hosted guide (multi-server vs. services) If you plan to run multiple replicas of the background services, make sure all replicas share the same PREFECT_SERVER_DOCKET_URL and Redis is reachable from each. Want me to review your compose/k8s manifests to double-check the env splits?
c
how to configure redis for API server
@Marvin - how to configure redis for API server
m
thought for 132.3 seconds
Got it. To enable Redis-backed features on the API server in Prefect 3.6.20, set the Redis-backed implementations for events/messaging, causal ordering, and concurrency lease storage. Do not set PREFECT_SERVER_DOCKET_URL on the API server (that’s only for background services). What to set on every API server instance
Copy code
# Required: install the integration
pip install prefect-redis

# Events/messaging stack (real-time updates, automations, etc.)
PREFECT_SERVER_EVENTS_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_SERVER_EVENTS_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering

# Deployment concurrency limits (lease storage)
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage

# Redis connection details used by the above
PREFECT_REDIS_MESSAGING_HOST=redis-host
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0
# If auth is required:
PREFECT_REDIS_MESSAGING_USERNAME=prefect_user
PREFECT_REDIS_MESSAGING_PASSWORD=****
# If TLS is required:
PREFECT_REDIS_MESSAGING_SSL=true
Notes - Use a different Redis DB (or separate Redis instance) for Docket on the background services to avoid key collisions (e.g., messaging on DB 0; Docket on DB 1). - Background services should also use the same messaging/ordering/lease storage settings as the API so events and concurrency are consistent across components, plus set PREFECT_SERVER_DOCKET_URL for the services only. - If you run multiple API server replicas, apply the same settings on each. Example split deployment (API only)
Copy code
PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@postgres:5432/prefect>
PREFECT_SERVER_API_HOST=0.0.0.0
PREFECT_SERVER_API_PORT=4200

# Redis-backed events + concurrency
PREFECT_SERVER_EVENTS_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_SERVER_EVENTS_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage
PREFECT_REDIS_MESSAGING_HOST=redis
PREFECT_REDIS_MESSAGING_DB=0

# UI points to your public API URL
PREFECT_UI_API_URL=<https://your.prefect.example.com/api>

# No Docket on the API server
# (do not set PREFECT_SERVER_DOCKET_URL here)
References - Messaging/ordering/lease storage modules live in the Redis integration: - prefect-redis integration (source) - Docket setting (services only): ServerDocketSettings - UI/API server settings: Server UI settings, API settings - Self-hosted/multi-server overview: Self-hosted guide If you can share your compose/k8s manifests, I can mark exactly where to add these and suggest Redis DB separation for messaging vs. Docket.
c
@Marvin - how to configure redis for API server and background services
m
thought for 60.4 seconds
Here’s a clear setup for Redis in a split deployment (API server and background services), on Prefect 3.6.20. Prereq (both API and services)
Copy code
# Install the Redis integration
pip install prefect-redis
# or: pip install "prefect[redis]"
Common Redis connection settings (used by both)
Copy code
# Connection details used by messaging/ordering/lease storage
PREFECT_REDIS_MESSAGING_HOST=redis-host
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0
# If auth/TLS:
PREFECT_REDIS_MESSAGING_USERNAME=prefect_user
PREFECT_REDIS_MESSAGING_PASSWORD=****
PREFECT_REDIS_MESSAGING_SSL=false
API server (no Docket here)
Copy code
# Required DB
PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@postgres:5432/prefect>

# Redis-backed events (real-time), ordering, and concurrency leases
PREFECT_SERVER_EVENTS_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_SERVER_EVENTS_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage

# Bind/UI (as needed)
PREFECT_SERVER_API_HOST=0.0.0.0
PREFECT_SERVER_API_PORT=4200
PREFECT_UI_API_URL=<https://your.public.prefect/api>

# Start API only
prefect server start --host 0.0.0.0 --port 4200 --no-services
Background services (add Docket)
Copy code
# Same DB and Redis-backed components as API
PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@postgres:5432/prefect>
PREFECT_SERVER_EVENTS_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_SERVER_EVENTS_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage
PREFECT_REDIS_MESSAGING_HOST=redis-host
PREFECT_REDIS_MESSAGING_DB=0

# Docket (services-only; coordinates scheduler/automations)
# Use a different Redis DB/index to avoid key collisions with messaging
PREFECT_SERVER_DOCKET_URL=<redis://redis-host:6379/1>
# or rediss://... and include username:password@ if required

# Start services
prefect server services start
Best practices - Use separate Redis DB indexes: DB 0 for messaging/order/leases, DB 1 for Docket. - Apply the same messaging/order/lease storage settings on both API and services so events/concurrency are consistent. - Do not set PREFECT_SERVER_DOCKET_URL on the API server. References - ServerDocketSettings (PREFECT_SERVER_DOCKET_URL) - prefect-redis integration (messaging/order/lease storage) - Self-hosted guide (split API/services) If you share your Compose/K8s manifests, I can mark exactly where to add these and suggest DB separation per component.
c
@Marvin - Are you sure DOCKET url is not required for API server. IN server UI settings, i see docket attributes
m
thought for 202.5 seconds
Short answer: Yes—no Docker URL is required for the Prefect API server. Assuming you’re on Prefect 3.x: - The server runs independently of Docker and does not need any Docker configuration to start or operate. - The Server UI Settings (the JSON from
/api/ui-settings
) expose things like
api_url
,
csrf_enabled
,
auth
, and feature
flags
. There isn’t a Docker URL required there. - If you’re seeing “docker” fields in the UI, they’re likely tied to deployment/worker configuration (e.g., Docker worker or image build settings), not the Server’s API settings. When a Docker URL is needed: - Only when using a Docker worker to execute flows in containers. In that case, you configure the worker to connect to the Docker daemon (e.g., via a DockerHost block’s
base_url
):
Copy code
from prefect_docker import DockerHost

docker_host = DockerHost(
    base_url="unix:///var/run/docker.sock"  # or "<tcp://127.0.0.1:2375>"
)
Could you share a screenshot or the exact field names you’re seeing in “Server UI settings”? If you can also confirm your Prefect version, I can be more specific. If you hit your server at
/api/ui-settings
, the JSON there will show exactly what the server is providing to the UI.