Short answer: A Prefect 3.x Process worker does not add meaningful overhead to GPU compute. Your flow runs in a child process on the same machine; Prefect mostly adds orchestration/logging around it. Low GPU utilization is usually caused by data/CPU/IO bottlenecks, process start-method issues, or competing processes rather than the worker itself. Here are the most common culprits and fixes: - Ensure only one flow uses a GPU at a time - If multiple flow runs land on the same machine/GPU, they’ll contend and often underutilize. Limit concurrency on the queue that feeds this worker to 1:

prefect work-queue set-concurrency-limit <queue-name> 1 --pool <work-pool-name>

- Or use a tag-based global limit if that fits better:

prefect concurrency-limit create gpu 1

Docs: Global concurrency limits - Pin the GPU device for the flow process - Set

CUDA_VISIBLE_DEVICES

for the deployment so the worker’s child process only sees the intended GPU:

python
  from prefect import flow

  @flow
  def my_flow(...):
      ...

  my_flow.from_source("<https://github.com/owner/repo>", entrypoint="path/to/file.py:my_flow").deploy(
      name="gpu-inference",
      work_pool_name="my-process-pool",
      job_variables={
          "env": {
              "CUDA_VISIBLE_DEVICES": "0",
              # optional tuning vars, if relevant to your setup:
              # "OMP_NUM_THREADS": "4",
              # "MKL_NUM_THREADS": "4",
          }
      },
  )

Docs: Customize job variables - Avoid returning large tensors from tasks - When a task returns big GPU/CPU tensors, Prefect will attempt to serialize/persist them which adds CPU/memory pressure. Prefer returning small summaries/paths and set:

python
  from prefect import task

  @task(persist_result=False, log_prints=False)
  def run_inference(...):
      ...

- Feed the GPU efficiently (PyTorch I/O/CPU pipeline) - Typical fixes for low utilization: - Increase

DataLoader(num_workers=4..8)

, use

pin_memory=True

, and set

prefetch_factor

appropriately. - Use

non_blocking=True

on tensor transfers and increase batch size (if memory permits). - For inference, consider

torch.backends.cudnn.benchmark = True

. - Reduce per-step Python overhead (heavy logging/prints, synchronous metrics, etc.). - Multiprocessing and start method with CUDA - If your workload uses multiprocessing (e.g.,

DataLoader

workers), ensure you’re not tripping over fork/spawn issues. Often safer to use spawn with CUDA:

python
  import torch.multiprocessing as mp
  mp.set_start_method("spawn", force=True)

- Also ensure any multiprocessing entry point is guarded (especially on Windows/macOS):

python
  if __name__ == "__main__":
      # launch flow or code that spawns worker processes

- Keep model/data-loader construction inside the task body so child workers don’t inherit half-initialized CUDA contexts. - Keep orchestration overhead low while the model runs - Default Prefect task runner is fine; you can also keep things simple/serial during the GPU section. - Minimize logging in tight loops, or set

PREFECT_LOGGING_LEVEL=WARNING

in the deployment env. - Sanity-check outside Prefect - Run the same code as a plain Python script and compare `nvidia-smi`/Nsight/PyTorch profiler. If utilization is low there too, the issue is not Prefect. If you need strict isolation or scheduling by GPU resources, consider containerized workers (e.g., Docker or Kubernetes workers with NVIDIA runtime) so each job gets a dedicated GPU resource. For Process workers, concurrency limits plus

CUDA_VISIBLE_DEVICES

are the key controls. Helpful links: - Workers overview

- Process worker API - Set env vars in job variables - Concurrency limits A few questions to help pinpoint the cause: - Are multiple flow runs sharing the same GPU at the same time? - What’s your

DataLoader

setup (batch size,

num_workers

,

pin_memory

,

prefetch_factor

)? - Are you returning large tensors from tasks? - OS and environment (bare metal, Conda, Docker), and are you setting

CUDA_VISIBLE_DEVICES

? - Roughly what utilization do you see when running the same script outside Prefect?