Short answer: yes. In Prefect 3.x you can run a Process-type work pool with a long‑running worker that continuously polls for flow runs—very similar to a traditional queue/worker service. What this looks like - Create a Process work pool, then start a worker that polls it indefinitely:

prefect work-pool create "my-process-pool" --type process
  prefect worker start --pool "my-process-pool" --name "$(hostname)" --limit 4

-

--limit

caps how many flow runs this worker will execute concurrently. - You can also set a pool-wide concurrency cap:

prefect work-pool set-concurrency-limit "my-process-pool" 10

- Run it as a production service (systemd example):

[Unit]
  Description=Prefect Process Worker
  After=network-online.target

  [Service]
  User=prefect
  WorkingDirectory=/opt/prefect
  Environment=PREFECT_API_URL=<https://api.prefect.cloud/api/accounts/<account-id>/workspaces/<workspace-id>>
  Environment=PREFECT_API_KEY=<your-api-key>
  # optional: Environment=PREFECT_LOGGING_LEVEL=INFO
  ExecStart=/usr/bin/env prefect worker start --pool my-process-pool --name %H --limit 4 --with-healthcheck
  Restart=always
  RestartSec=5
  # If you're using a venv:
  # ExecStart=/opt/prefect/.venv/bin/prefect worker start --pool my-process-pool --name %H --limit 4 --with-healthcheck

  [Install]
  WantedBy=multi-user.target

Then:

sudo systemctl daemon-reload
  sudo systemctl enable prefect-worker
  sudo systemctl start prefect-worker

-

--with-healthcheck

exposes a simple /health endpoint for external monitoring. - Alternatively, run the worker in a container as a long-lived service (e.g., Docker/Compose/Kubernetes), but note that with a Process work pool, each flow still runs as a subprocess inside that same long-lived container. Key considerations and caveats - Isolation and reproducibility: Process workers run flow runs as local subprocesses. There’s no per-run container/image isolation. Make sure your code and dependencies are present on the host (or use deployments with

flow.from_source(...)

so code is pulled at run time). - Scaling: You can run multiple process workers (even on different machines) pointing to the same pool to increase throughput, but each machine must have the right environment available. There’s no per-run provisioning like Docker/Kubernetes. - Resource controls: No per-run CPU/memory limits. Use

--limit

and pool concurrency to avoid resource contention and monitor host resources closely. - Reliability: Use a service manager (systemd, Supervisor, or an orchestrator) with auto-restart. The healthcheck endpoint helps liveness/readiness probes and external monitoring. Since you work with Vertex AI work pools: Process pools are a good fit for “always-on queue” on a single host or a small fleet, especially when Docker images aren’t a great match. If you need per-run isolation or strict resource constraints, Docker/Kubernetes pools are better options. If you want serverless execution, keep leaning on your Vertex AI pool for those runs, and reserve the Process pool for simpler, host-based workloads. Docs and references - Work pools: Work pools - Workers: Workers overview - Healthchecks: Worker healthchecks - Daemonizing workers (systemd etc.): Daemonize processes - Default Process job template: Process worker API Would you prefer to run this via systemd on a VM, inside a long-lived container, or something else? Also, how many concurrent runs do you expect and how are you distributing code (pre-installed vs

from_source

)? I can tailor a config for your setup.

Yes—running a Process work pool worker on Cloud Run works well if you configure the service to stay warm and give the worker CPU to poll continuously. What you need on Cloud Run - Keep at least one instance running: set min instances >= 1. - Ensure background polling runs: enable “CPU always allocated”. - Expose an HTTP port: start the worker with

--with-healthcheck

and bind to

$PORT

so Cloud Run sees the container as healthy. - Provide Prefect credentials via env vars:

PREFECT_API_URL

and

PREFECT_API_KEY

. A minimal container setup Dockerfile:

FROM python:3.11-slim

# Install system deps if your flows need them
# RUN apt-get update && apt-get install -y ...

# Install Prefect (add extras as needed, e.g., [gcp], [aws], etc.)
RUN pip install --no-cache-dir prefect

# If you want code baked in, copy it; otherwise use flow.from_source(...) in deployments
# COPY . /app
WORKDIR /app

# Simple entrypoint that binds the healthcheck to Cloud Run's PORT
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

# Cloud Run will send traffic to $PORT; we use it for the worker healthcheck server
ENV PREFECT_WORKER_WEBSERVER_HOST=0.0.0.0

# Pool name, worker name, and concurrency can be overridden at deploy time
ENV PREFECT_WORK_POOL=my-process-pool
ENV WORKER_NAME=cloud-run
ENV WORKER_LIMIT=2

CMD ["/entrypoint.sh"]

entrypoint.sh:

#!/bin/sh
# Bind the healthcheck server to Cloud Run's PORT
export PREFECT_WORKER_WEBSERVER_PORT="${PORT:-8080}"

# Fail fast if credentials are missing
: "${PREFECT_API_URL:?Missing PREFECT_API_URL}"
: "${PREFECT_API_KEY:?Missing PREFECT_API_KEY}"

# Start the worker; it will poll forever
exec prefect worker start \
  --pool "${PREFECT_WORK_POOL}" \
  --name "${WORKER_NAME}-$(hostname)" \
  --limit "${WORKER_LIMIT}" \
  --with-healthcheck

Deploy to Cloud Run (example) - Build and deploy (set your values accordingly):

gcloud builds submit --tag <http://gcr.io/<PROJECT>/<SERVICE_NAME>:latest|gcr.io/<PROJECT>/<SERVICE_NAME>:latest>

gcloud run deploy <SERVICE_NAME> \
  --image <http://gcr.io/<PROJECT>/<SERVICE_NAME>:latest|gcr.io/<PROJECT>/<SERVICE_NAME>:latest> \
  --region <REGION> \
  --platform managed \
  --min-instances 1 \
  --cpu 1 \
  --memory 1Gi \
  --max-instances 1 \
  --set-env-vars PREFECT_API_URL=<https://api.prefect.cloud/api/accounts/<account-id>/workspaces/<workspace-id>> \
  --set-secrets PREFECT_API_KEY=PREFECT_API_KEY:latest \
  --set-env-vars PREFECT_WORK_POOL=my-process-pool,WORKER_LIMIT=4 \
  --allow-unauthenticated=false

- In Cloud Run: - Enable “CPU is always allocated”. - Optionally configure a health check hitting

/health

(the worker will serve it). - Use a service account with access to any external resources your flows require (e.g., GCS, BigQuery). How this behaves - The worker runs indefinitely, polling your Process pool and launching flows as subprocesses inside the same container. - You can scale concurrency with

--limit

or add more instances (set min instances > 1) to run more flows in parallel. Note: Cloud Run does not auto-scale based on Prefect queue length; you’ll scale instance counts yourself or via a custom scaler. - There’s no per-run isolation (unlike Docker/Kubernetes pools); ensure all dependencies are present in the container or use

flow.from_source(...)

to pull code at runtime. Caveats and tips - Cloud Run instances can be restarted at any time; keep flows idempotent and set retry policies. - If you need strong isolation or per-run resource limits, consider a Docker or Kubernetes work pool instead. - If you want “scale-to-zero” and event-driven execution, you may prefer a push/serverless work pool (you mentioned Vertex AI—great option for that). Useful Prefect docs - Work pools: Work pools - Workers: Workers overview - Worker healthchecks: Worker healthchecks

If you share your expected concurrency and how you package your flow code (pre-baked vs

from_source

), I can tailor the Dockerfile and flags further.

Short answer: it depends on the work pool you’re targeting. - If you’re using a Process work pool (pull model): Yes, you can run a process worker inside that container. It just needs Prefect and network access to your Prefect API. The worker will poll the pool and run flows in subprocesses on that machine. In this setup, it’s typical to fetch flow code at runtime via

Flow.from_source(...)

, so baking code into the worker image is optional. - Start it with:

prefect worker start -p <your-process-pool-name>

- Docs: Run flows with a Process work pool, Workers overview - If you’re using a Vertex AI work pool (push model): Do not run a process worker. Push pools don’t use a long-running worker; Prefect submits your container image directly to Vertex AI. In this case, your container (with flow code + Prefect installed) should be used as the runtime image for the deployment, not as a worker. - Create your deployment pointing at your source and image (build/push as needed), and target your Vertex AI work pool. For example:

from prefect.flows import Flow

    flow = Flow.from_source(
        "<https://github.com/your-org/your-repo>",
        "flows/my_flow.py:my_flow",
    )

    flow.deploy(
        name="prod",
        work_pool_name="vertex-ai-pool",
        image="us-docker.pkg.dev/<gcp-project>/<repo>/<image>:<tag>",
        # set build/push True if Prefect should build/push; False if you prebuilt the image
        build=False,
        push=False,
    )

- Docs: Serverless (push) work pools, GCP (Cloud Run & Vertex AI) guide Tips: - For Process pools, ensure the container has any tools needed to pull code (e.g., git) and that

PREFECT_API_URL

and

PREFECT_API_KEY

are set if using Prefect Cloud. - For Vertex AI, make sure your image is compatible with Vertex AI (generally linux/amd64) and includes all dependencies and your flow code; no separate worker is required. Given you work with Vertex AI work pools, I’d recommend using your container as the runtime image for the deployment and skipping a process worker entirely. Does that match your setup, or are you specifically targeting a Process work pool on a fixed machine?

Short version: You can do it, but Cloud Run isn’t an ideal place for a long‑running polling worker unless you configure it carefully. Cloud Run expects an HTTP server and will scale to zero/kill CPU between requests unless you enable “CPU is always allocated.” A Prefect process worker is a long‑running poller. If you still want to host it on Cloud Run so flows run “in place” (no new infra per run), use this pattern: - Create a Process work pool

prefect work-pool create "my-process-pool" --type process

- Build a container that: - Has Prefect installed (and any flow deps) - Starts a tiny HTTP server for Cloud Run health checks - Starts the Prefect worker in the background, polling your process pool Example run script:

#!/usr/bin/env bash
set -euo pipefail

# 1) Start the Prefect worker in the background
prefect worker start -p "my-process-pool" &

# 2) Minimal HTTP server to satisfy Cloud Run
python -c "import http.server, socketserver, os; socketserver.TCPServer(('', int(os.environ.get('PORT','8080'))), http.server.SimpleHTTPRequestHandler).serve_forever()"

Key Cloud Run settings to make this stable: - Set min instances to 1 (keeps a worker alive) - Enable “CPU is always allocated” so the worker can poll even when there are no requests - Configure

PREFECT_API_URL

and

PREFECT_API_KEY

as env vars/secrets - Ensure your image is built for linux/amd64 (to avoid arch mismatches) - Consider limiting concurrency either at the work pool/queue level in Prefect or by only running a single worker per Cloud Run instance so flow runs don’t fight for the same CPU/memory Flow code options: - For a process pool, it’s common to fetch code at runtime via

Flow.from_source(...)

so the container doesn’t have to include all flow code. Alternatively, bake your code into the image if you prefer. Caveats: - Cloud Run is not designed for headless background processes. This pattern works but is “hacky.” If you ever want serverless autoscaling per run instead, use a Cloud Run push work pool (no worker needed, Prefect submits jobs directly). But that does spin up per‑run infrastructure. - Restarts/revisions will interrupt the worker; ensure your flows are retry‑safe. Docs for reference: - Workers overview - Process work pool guide - Store/pull flow code with from_source - If you decide to switch to serverless/push later: Serverless (push) work pools Does this approach (min instances + CPU always allocated + tiny HTTP server + background Prefect worker) fit your constraints? If yes, I can share a full Dockerfile and

gcloud run deploy

example tailored to your environment.

Short answer: running a Docker worker inside Cloud Run won’t work. The Docker worker needs access to a Docker daemon (the Docker socket) to start sibling containers for each flow run. Cloud Run is fully managed and does not allow privileged access or Docker-in-Docker, so the worker cannot launch containers there. Good alternatives depending on your goal: - Keep everything in one always-on service (no new infra per run): Use a Process work pool with a long‑lived worker in Cloud Run. This runs flows directly in the service process. Configure: - Min instances = 1 - CPU always allocated - Tiny HTTP server for health checks + background

prefect worker start -p <process-pool>

- Fetch code at runtime via

Flow.from_source(...)

or bake it into the image Docs: Process work pool guide, Workers overview - If you actually want per‑run containerized isolation on GCP: Use a Cloud Run push work pool. Prefect will submit each flow run as its own Cloud Run job/service from a container image that includes your flow code, with no worker process to manage. Docs: Serverless (push) work pools, GCP worker guide - If you want a Docker worker (pull model) specifically: Run it where Docker is available, e.g., a small GCE VM or on GKE. That worker can then launch per‑run containers. Docs: Docker work pool guide Given your “no new infra per run” goal, the Process worker on Cloud Run is the closest fit. If you want, I can share a minimal Dockerfile and entrypoint script to run the Prefect worker in the background alongside a tiny HTTP server, plus the Cloud Run settings to keep it stable.

Short answer: you can’t run a Process worker as a GCP-managed “worker pool” (push). A Process worker is a long‑running poller you must host yourself. If you want “like Cloud Run but without healthchecks,” the simplest is a small Compute Engine VM (or a GKE Deployment) running the worker. Two solid options on GCP without HTTP healthchecks: - Compute Engine VM (recommended for simplicity) 1) Create a Process work pool:

prefect work-pool create "process-pool" --type process

2) On a small VM (e.g., e2-small), install Python/Prefect and set env vars: -

PREFECT_API_URL

and

PREFECT_API_KEY

3) Run as a systemd service so it stays up across reboots:

# /etc/systemd/system/prefect-worker.service
     [Unit]
     Description=Prefect Process Worker
     After=network-online.target
     Wants=network-online.target

     [Service]
     Environment=PREFECT_API_URL=<https://api.prefect.cloud/api/accounts/<acct>/workspaces/<ws>>
     Environment=PREFECT_API_KEY=<your-key>
     ExecStart=/usr/local/bin/prefect worker start -p process-pool --name gcp-vm-1
     Restart=always
     RestartSec=10

     [Install]
     WantedBy=multi-user.target

Then:

sudo systemctl daemon-reload
     sudo systemctl enable --now prefect-worker

4) Flow code: either fetch at runtime via

Flow.from_source(...)

or bake it into the VM image/container you use on the VM. - GKE (if you already have Kubernetes) - Run a Deployment with your Prefect image and

command: ["prefect","worker","start","-p","process-pool"]

. Kubernetes will keep it alive; no HTTP server needed. Notes and tips: - Process pools run flows directly in the worker’s process space; no per‑run infra is created. - Set Prefect concurrency limits on the work pool/queues if you want to cap parallelism. - Make sure your environment matches the CPU arch you deploy on (linux/amd64). - If you need per‑run container isolation later, switch to Cloud Run push or Vertex AI push pools (no worker), but those do spin up infra per run. Docs: - Workers overview: Workers overview - Process work pool: Run flows in local processes - Storing/pulling flow code: Store and fetch flow code If you want, I can share a GKE Deployment snippet or a startup script for a GCE VM image with Prefect preinstalled. Which route do you prefer (GCE VM vs GKE)?