Short answer: your proposal is not an antipattern. For your workload, a process work pool with one or more process workers is a perfectly valid (and common) production setup in Prefect 3.x—especially for lightweight, I/O-bound flows where a pod-per-run is cost-prohibitive. Why a process worker can be a good fit - Minimal overhead: A single worker pod can execute many flow runs as subprocesses instead of creating a Pod per run. This typically reduces memory/CPU overhead vs k8s worker jobs (which incur ~hundreds of MB per pod in your case). - Full orchestration: You still get deployments, scheduling, retries, concurrency limits, tags, and automations. - Simple scaling: Scale by adding more worker replicas or adjusting pool/queue concurrency. Trade-offs to be aware of - Failure/Isolation: A process worker is a single pod; if it restarts, you lose all runs it was executing. You don’t get per-run container isolation. If you need strict isolation or specific per-run dependencies, Kubernetes/Docker workers are better. - Dependency conflicts: Runs share the same base environment. If your flows have divergent dependencies, containerized workers are safer. - Resource caps per worker: Put reasonable concurrency limits in place to avoid OOM on the worker pod. What to use for concurrency and rate-limiting (Prefect 3.x) - Work pool concurrency limit: Caps concurrent flow runs started from a pool.

prefect work-pool create "io-process-pool" --type process
  prefect work-pool set-concurrency-limit "io-process-pool" 20

- Work queue concurrency limit: Optional per-queue cap if you segment runs by queue.

prefect work-queue create "http-queue" --pool "io-process-pool"
  prefect work-queue set-concurrency-limit "http-queue" 10

- Tag-based concurrency limits (task-level): Create a limit for a tag, then tag tasks to gate concurrent executions across all flows.

prefect concurrency-limit create external-api 10

Then tag tasks or use the concurrency contexts below. - Code-based concurrency gates and rate limits: Use Prefect’s concurrency contexts to enforce global limits/rates from code (sync or asyncio):

from prefect.concurrency.sync import concurrency
  from prefect import flow, task
  import httpx

  @task(tags=["external-api"])  # optional if you also want tag-based visibility/controls
  def call_api(url):
      with concurrency("external-api", occupy=1, timeout_seconds=30):
          return httpx.get(url, timeout=60).json()

  @flow
  def run_many(urls):
      return [call_api.submit(u) for u in urls]

Or rate-limit (async example):

from prefect.concurrency.asyncio import rate_limit

  async with rate_limit("external-api-qps", occupy=1):
      # make an async HTTP call here
      ...

About background tasks - Background tasks are useful for lightweight concurrent work inside one flow run, but they do not give you per-run orchestration controls (scheduling, retries across runs, etc.). If you need cross-run concurrency control and operational guarantees, stick with “normal” flows + workers and use the concurrency primitives above. - You can still use concurrency contexts inside background tasks if you go that route, but most teams prefer regular flows for the kind of job orchestration you described. Starting a process worker - Run a process worker that polls your pool (use a Deployment/StatefulSet in your cluster so it’s supervised):

prefect worker start -p "io-process-pool" --name io-worker-1

- Scale by: - Increasing the work pool/queue concurrency limit(s) - Running multiple worker replicas - Adjusting the pod’s CPU/RAM requests/limits When to prefer Kubernetes workers - You need one flow per isolated container (e.g., strict per-run dep isolation, different base images, GPU, job-specific node scheduling, etc.) - You’re okay paying for the per-run pod overhead for those isolation benefits.

Given your description (simple flows that make long HTTP calls), a process pool is a great way to reduce cost and keep the orchestration benefits. Many users run process workers in production for exactly this pattern. If you want, share how you’re deploying workers today (via the Prefect Helm chart or your own Deployment YAML), and your target concurrency (e.g., total concurrent HTTP requests, QPS per API, memory budget per worker). I can suggest concrete values.yaml changes or a Deployment manifest that matches your goals. Reference you already found: https://docs.prefect.io/v3/advanced/background-tasks

We are planning to deploy another worker of type kubernetes for some workflows that we want to be executed in their own pod