<@ULVA73B9P> How to deploy on kubernetes the backg...
# marvin-ai
l
@Marvin How to deploy on kubernetes the background-services separately from the server for production environments where we have multiple redis instances configured with sentinel e.g. PREFECT_REDIS_CONNECTION_URL: sentinel://blabla@mymaster?service=mymaster, REFECT_REDIS_SENTINEL_HOSTS: prefect-server-redis-node-0.prefect-server-redis-headless.prefect-server.svc.cluster.local26379,prefect server redis node 1.prefect server redis headless.prefect server.svc.cluster.local26379?
m
thought for 672.9 seconds
Short answer - Yes: you can run the API and the background services as separate Kubernetes pods. - No: Prefect 3 does not support Redis Sentinel URLs or sentinel host lists. The
sentinel://...
URL and env vars like
PREFECT_REDIS_CONNECTION_URL
/
PREFECT_REDIS_SENTINEL_HOSTS
are not recognized by Prefect 3. - Workaround: put a sentinel-aware proxy (HAProxy/Envoy/redis-sentinel-proxy or a managed Redis endpoint that handles failover) in front of Redis, then point Prefect at the single stable
host:port
. Key commands - API only (no background services):
Copy code
prefect server start --no-services --host 0.0.0.0 --port 4200
- Background services only:
Copy code
prefect server services start
Kubernetes example (two Deployments) API deployment (scale this)
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-server-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: prefect-server-api
  template:
    metadata:
      labels:
        app: prefect-server-api
    spec:
      containers:
      - name: api
        image: prefecthq/prefect:3-latest
        # IMPORTANT: ensure your image has the redis extra if you’ll use Redis-backed features
        # e.g., build from prefecthq/prefect:3-latest and `pip install "prefect[redis]"`
        command: ["prefect", "server", "start", "--no-services", "--host", "0.0.0.0", "--port", "4200"]
        ports:
        - name: api
          containerPort: 4200
        env:
        # Postgres (required)
        - name: PREFECT_SERVER_DATABASE_CONNECTION_URL
          valueFrom:
            secretKeyRef:
              name: prefect-db
              key: connection-url
        # Basic server settings
        - name: PREFECT_SERVER_API_HOST
          value: "0.0.0.0"
        - name: PREFECT_SERVER_API_PORT
          value: "4200"
        - name: PREFECT_SERVER_LOGGING_LEVEL
          value: "INFO"
        # If you front Redis with a stable proxy endpoint, add the same Redis env here if needed
        # - name: PREFECT_MESSAGING_BROKER
        #   value: "prefect_redis.messaging"
        # - name: PREFECT_MESSAGING_CACHE
        #   value: "prefect_redis.messaging"
        # - name: PREFECT_SERVER_EVENTS_CAUSAL_ORDERING
        #   value: "prefect_redis.ordering"
        # - name: PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE
        #   value: "prefect_redis.lease_storage"
        # - name: PREFECT_REDIS_MESSAGING_HOST
        #   value: "redis-proxy.default.svc.cluster.local"
        # - name: PREFECT_REDIS_MESSAGING_PORT
        #   value: "6379"
        # - name: PREFECT_REDIS_MESSAGING_DB
        #   value: "0"
        readinessProbe:
          httpGet:
            path: /api/health
            port: 4200
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /api/health
            port: 4200
          initialDelaySeconds: 30
          periodSeconds: 10
Background services deployment (run 1 replica) ``` apiVersion: apps/v1 kind: Deployment metadata: name: prefect-server-services spec: replicas: 1 selector: matchLabels: app: prefect-server-services template: metadata: labels: app: prefect-server-services spec: containers: - name: services image: prefecthq/prefect:3-latest # ensure the image includes the redis extra: pip install "prefect[redis]" command: ["prefect", "server", "services", "start"] env: # Postgres (required) - name: PREFECT_SERVER_DATABASE_CONNECTION_URL valueFrom: secretKeyRef: name: prefect-db key: connection-url # Use Redis-backed implementations - name: PREFECT_MESSAGING_BROKER value: "prefect_redis.messaging" - name: PREFECT_MESSAGING_CACHE value: "prefect_redis.messaging" - name: PREFECT_SERVER_EVENTS_CAUSAL_ORDERING value: "prefect_redis.ordering"
- name: PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE value: "prefect_redis.lease_storage" # Connect to a single stable Redis endpoint (via sentinel-aware proxy) - name: PREFECT_REDIS_MESSAGING_HOST value: "redis-proxy.default.svc.cluster.local" - name: PREFECT_REDIS_MESSAGING_PORT value: "6379" - name: PREFECT_REDIS_MESSAGING_DB value: "0" # optional auth/TLS # - name: PREFECT_REDIS_MESSAGING_USERNAME # value: "default" # - name: PREFECT_REDIS_MESSAGING_PASSWORD # valueFrom: # secretKeyRef: # name: redis-auth # key: password # - name: PREFECT_REDIS_MESSAGING_SSL # value: "true" ``` Why your sentinel env won’t work - Prefect 3’s Redis integration (prefect-redis) expects single-node connection parameters: -
PREFECT_REDIS_MESSAGING_HOST
,
PREFECT_REDIS_MESSAGING_PORT
,
PREFECT_REDIS_MESSAGING_DB
,
PREFECT_REDIS_MESSAGING_USERNAME
,
PREFECT_REDIS_MESSAGING_PASSWORD
,
PREFECT_REDIS_MESSAGING_SSL
- It does not parse
sentinel://...
URLs and does not accept
PREFECT_REDIS_SENTINEL_HOSTS
. - Source code for the Redis client and settings: prefect-redis integration and CLI/server commands: server CLI Suggested Sentinel approaches - Put a sentinel-aware proxy in front of Redis (HAProxy/Envoy/redis-sentinel-proxy) and expose a stable
<redis://host:port>
. Point Prefect at that endpoint with the
PREFECT_REDIS_MESSAGING_*
vars shown above. - Use a managed Redis offering that presents a stable primary endpoint during failover. Quick checklist - Build an image with
prefect[redis]
installed for both the API and services pods if you use Redis-backed features. - API: run
prefect server start --no-services
, set
PREFECT_SERVER_API_HOST=0.0.0.0
. - Services: run
prefect server services start
, set Postgres and Redis env vars as above. - Run 1 replica for services (they coordinate work; multiple replicas aren’t typically necessary). If a sentinel-aware proxy is acceptable, I can share a minimal HAProxy config example. Also happy to adapt the YAML to your exact namespaces, image, and secrets.