Great question — and you’re right that the docs are terse here. Here’s what’s going on and why you’re seeing this now. Summary - Work pool storage is a pool-level setting that tells Prefect where to upload/run “flow bundles” when you use the experimental dynamic-infrastructure decorators like @kubernetes. - It’s required now because these decorators package your flow code and dependencies into a bundle and run it on remote infrastructure; the worker needs a shared, cloud-accessible place to fetch that bundle. - Previously, with deployments or images that already contained your code, this pool-level storage wasn’t needed. What “work pool storage” means - It’s a storage configuration attached to a work pool that: - Defines how to upload bundles (e.g., to S3/GCS/Azure) - Defines how workers download/execute those bundles - Creates/links a default result storage for runs launched by that pool - Configure it once per pool with:

prefect work-pool storage configure s3 <POOL_NAME> ...
  prefect work-pool storage configure gcs <POOL_NAME> ...
  prefect work-pool storage configure azure-blob-storage <POOL_NAME> ...

- CLI reference: prefect work-pool storage Why it’s needed now (and wasn’t before) - Before (e.g., with deployments and/or custom images), your flow’s source was either: - Pulled from a repo via deployment storage (

flow.from_source(...).deploy(...)

), or - Already baked into the container image you ran. - With the experimental @kubernetes path, Prefect serializes your flow + local imports (+ optional include_files) into a bundle and: - Uploads it to the pool’s configured storage - Starts a Kubernetes job whose first step is to download and execute that bundle - Because the worker pod needs a consistent place to fetch that bundle, the pool must have storage configured. What happens under the hood with @kubernetes - When you call the decorated flow for remote execution via a work pool: 1) Prefect creates a bundle (your flow function + local modules + optional include_files). 2) Uploads that bundle to the pool’s storage (S3/GCS/Azure). 3) The Kubernetes job uses a Prefect image, downloads the bundle from that storage, installs any needed deps, and runs the flow. - No volume mounts of your source are required; your code is shipped as the bundle. - More background: Submit flows directly to dynamic infrastructure When it’s required - Required when you run via a work pool (e.g., passing

work_pool="work-gpu"

on the decorator or using

.submit_to_work_pool()

). - Not required for: - Local/in-process execution (calling the flow like a normal function) - Classic deployments that use

flow.from_source(...).deploy(...)

(code is sourced by the deployment), or - Images that already contain your code (typical deployment pattern) How to configure - S3 example:

# Create or reuse an AWS credentials block first (name shown here as my-aws-creds)
  prefect work-pool storage configure s3 work-gpu \
    --bucket my-prefect-bundles \
    --aws-credentials-block-name my-aws-creds

- GCS example:

prefect work-pool storage configure gcs work-gpu \
    --bucket my-prefect-bundles \
    --service-account-info-block-name my-gcp-creds

- Inspect what’s set:

prefect work-pool storage inspect work-gpu

If you share which cloud you’re using (S3, GCS, Azure) and how you were running flows before (deployments vs. custom images), I can give you the exact command(s) to run and help you migrate smoothly.

Correct — using

@flow(result_storage=S3Block)

is not enough. - Flow

result_storage

controls where your flow’s outputs are persisted. - Work pool storage controls where the code bundle (your flow + local deps + optional files) is uploaded so the Kubernetes worker/pod can download and execute it. With the experimental @kubernetes path, when you run via a work pool (e.g.,

work_pool="work-gpu"

or

.submit_to_work_pool()

), Prefect must upload a bundle to pool storage and then the pod fetches it. Even if you set

result_storage

on the flow, the worker still needs pool storage for the bundle itself. Notes: - The pool storage config also wires up a default result storage for runs from that pool. Your

@flow(result_storage=...)

can override where results land, but it does not replace the bundle upload/download steps. - Pool storage is not required if you run locally/in-process or use other patterns (e.g., classic deployments with

flow.from_source(...).deploy(...)

, or direct

.submit()

to a cluster you can access). It is required for the experimental work-pool-backed path. CLI reference and docs: - Work pool storage commands: prefect work-pool storage - Background on direct-to-infrastructure execution: Submit flows directly to dynamic infrastructure

Short answer: there’s no 1:1 equivalent in Prefect 2.x. Closest concepts in 2.x: - Deployment storage (per-deployment): You’d attach a storage block (e.g., GitHub/S3/GCS) to the deployment so the agent/worker could pull your code at run time. Alternatively, you’d bake your code into the container image. There was no pool-level storage setting. - Infrastructure blocks/templates (per-deployment or pool default): Defined how/where the job ran (Kubernetes, Docker, ECS), sometimes with “pull steps” to fetch code. Again, code delivery lived with the deployment, not the pool. - Result storage blocks: Controlled where results were persisted, just like

@flow(result_storage=...)

does in 3.x. What’s new in 3.x with the experimental decorators: - Pool-level “work pool storage” is specifically for the bundle-based execution path used by decorators like

@kubernetes

. Prefect packages your flow + local deps/files into a bundle, uploads it to the pool’s configured storage, and the worker/pod downloads and executes it. - This pool-scoped storage requirement didn’t exist in 2.x because code delivery was handled per-deployment (storage blocks) or via images, not via a shared bundle store at the pool level. So in 2.x you would have: - Used deployment storage blocks or images to get code onto the remote runner - No need to configure a pool-level storage target In 3.x (experimental @kubernetes path), you now configure pool storage once so any flow submitted to that pool can upload its bundle and be executed by the worker.

By default, the experimental @kubernetes decorator (via the Kubernetes worker) uses Prefect’s default image resolved by

prefect.utilities.dockerutils.get_prefect_image_name()

. What that resolves to: - Production installs:

prefecthq/prefect:<your_prefect_version>-python<your_python_minor>

- Example:

prefecthq/prefect:3.1.2-python3.11

- Dev/nightly installs:

prefecthq/prefect-dev:sha-<7char_git_sha>-python<minor>

- Example:

prefecthq/prefect-dev:sha-ab12cd3-python3.11

How to check on your machine:

from prefect.utilities.dockerutils import get_prefect_image_name
print(get_prefect_image_name())

How to override: - Directly on the decorator:

from prefect import flow
  from prefect_kubernetes.experimental import kubernetes

  @kubernetes(work_pool="work-gpu", image="my-registry/my-prefect-cuda:latest")
  @flow
  def my_flow():
      ...

- Or set a base job template on the work pool with your desired image. Note: For GPU workloads, you’ll typically want to specify your own CUDA-enabled image; the default Prefect images are CPU-only.

I couldn’t find any examples in our repos that show the experimental @kubernetes decorator using an explicit image= argument. The decorator absolutely supports it though — any keyword you pass becomes a job variable on the Kubernetes worker, and image is one of those variables. References - Decorator source: https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-kubernetes/prefect_kubernetes/experimental/decorators.py - Worker variables (includes image): https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-kubernetes/prefect_kubernetes/worker.py - How to customize infra via decorators: Docs section Minimal example you can use today

from prefect import flow
from prefect_kubernetes.experimental import kubernetes

@kubernetes(
    work_pool="my-k8s-pool",
    image="my-registry/prefect-cuda:12.2-runtime"  # your custom image here
)
@flow
def my_flow():
    import platform
    return f"Running on: {platform.platform()}"

# Local submission to a work pool (requires worker running)
my_flow.submit_to_work_pool()

You can combine image with other job variables (namespace, resources, env, etc.) the same way:

@kubernetes(
    work_pool="my-k8s-pool",
    image="registry/my-image:latest",
    namespace="ml",
    cpu="2",
    memory="8Gi",
    env={"EXTRA_FLAG": "1"},
)
@flow
def train():
    ...

If you’d like, tell me your registry and resource needs (e.g., GPU, CUDA version), and I’ll tailor an example manifest and image suggestion.

Short answer: yes, it works with a local ephemeral server — but you still need a work pool with storage configured. What “works locally” means with @kubernetes - Direct call or .submit: Prefect will spin up a local Kubernetes worker process for you, package your flow as a bundle, upload it to the work pool’s storage, and create a K8s job. This requires: - A Kubernetes work pool that exists - Work pool storage configured (S3/GCS/Azure) - Access to your K8s cluster from your machine - .submit_to_work_pool: Same bundle/upload + flow run creation, but it does not spin up a local worker; you need a worker already running for that pool. Ephemeral server specifics - You do not need to run a separate Prefect server. If you haven’t set an API URL, Prefect will run an in-process ephemeral API. All the same requirements for pool + storage still apply. - You can create the work pool via CLI or UI, then configure storage:

prefect work-pool create my-k8s-pool --type kubernetes
  prefect work-pool storage configure s3 my-k8s-pool --bucket my-bundles --aws-credentials-block-name my-aws-creds

- Optional: Start a worker yourself (useful for .submit_to_work_pool):

prefect worker start --pool my-k8s-pool

Why storage is still required locally - The experimental path always uses bundles for remote execution. Even with an ephemeral server, the pod must download your code bundle from cloud storage; that’s what the pool storage config enables. If you want, tell me which cloud you’re using (S3/GCS/Azure), and I’ll give you exact commands to set up the pool storage for your local ephemeral setup.

Not necessarily. You don’t need Minikube specifically — you just need access to any Kubernetes cluster that the Prefect Kubernetes worker can talk to. Common options: - Local clusters: Minikube, kind (Kubernetes in Docker), Docker Desktop Kubernetes, Rancher Desktop, MicroK8s - Remote clusters: EKS, GKE, AKS, on-prem Key points to make it work: - The worker (run locally or in-cluster) must have Kubernetes credentials (kubeconfig or in-cluster service account) and RBAC to create Jobs in your target namespace. - Your cluster must be able to pull the image you specify (or the default Prefect image from Docker Hub). - Your pods must have network access to the work pool storage you configured (S3/GCS/Azure) to download the bundle and write results. If you need GPU (you mentioned a work-gpu pool), be aware: - Many local options don’t support GPU easily. You’ll typically want a cluster with NVIDIA GPU nodes and the NVIDIA device plugin installed (e.g., EKS/GKE/AKS GPU nodes). On Linux you can make MicroK8s/kind work with NVIDIA runtime, but it’s more involved. If you tell me your OS and whether you need GPU, I can recommend the quickest cluster setup path and any extra steps you’ll need.

Short answer: not today via Terraform alone. - There is a Prefect Terraform provider, but it does not yet support configuring “work pool storage” (the bundle upload/execution steps) for a pool. - You can manage this reliably via the Prefect API or Python client (the same flow the CLI uses). What you can do now - Pre-create the S3 credential and result storage blocks via API/Python (or the CLI), then set the work pool’s storage configuration via the API. - This is exactly what

prefect work-pool storage configure s3 ...

automates. Outline of an automated setup (Python client)

from prefect.client.orchestration import get_client
from prefect.client.schemas.actions import WorkPoolUpdate, BlockDocumentCreate, BlockDocumentUpdate
from prefect.client.schemas.objects import WorkPoolStorageConfiguration
from prefect.exceptions import ObjectNotFound

POOL = "work-gpu"
BUCKET = "my-prefect-bundles"
AWS_CREDS_BLOCK = "my-aws-creds"
RESULT_BLOCK_NAME = f"default-{POOL}-result-storage"

async def configure_pool_storage():
    async with get_client() as client:
        # 1) Ensure result storage block exists/updated
        try:
            result_block = await client.read_block_document_by_name(
                name=RESULT_BLOCK_NAME, block_type_slug="s3-bucket"
            )
            await client.update_block_document(
                block_document_id=result_block.id,
                block_document=BlockDocumentUpdate(
                    data={
                        "bucket_name": BUCKET,
                        "credentials": {"$ref": {"block_document_name": AWS_CREDS_BLOCK}},
                    }
                ),
            )
        except ObjectNotFound:
            block_type = await client.read_block_type_by_slug("s3-bucket")
            schema = await client.get_most_recent_block_schema_for_block_type(block_type.id)
            result_block = await client.create_block_document(
                block_document=BlockDocumentCreate(
                    name=RESULT_BLOCK_NAME,
                    block_type_id=block_type.id,
                    block_schema_id=schema.id,
                    data={
                        "bucket_name": BUCKET,
                        "credentials": {"$ref": {"block_document_name": AWS_CREDS_BLOCK}},
                    },
                )
            )

        # 2) Update the work pool with bundle upload/execute steps
        await client.update_work_pool(
            work_pool_name=POOL,
            work_pool=WorkPoolUpdate(
                storage_configuration=WorkPoolStorageConfiguration(
                    bundle_upload_step={
                        "prefect_aws.experimental.bundles.upload": {
                            "requires": "prefect-aws",
                            "bucket": BUCKET,
                            "aws_credentials_block_name": AWS_CREDS_BLOCK,
                        }
                    },
                    bundle_execution_step={
                        "prefect_aws.experimental.bundles.execute": {
                            "requires": "prefect-aws",
                            "bucket": BUCKET,
                            "aws_credentials_block_name": AWS_CREDS_BLOCK,
                        }
                    },
                    default_result_storage_block_id=result_block.id,
                )
            ),
        )

If you’re committed to Terraform-only workflows: - You can still use Terraform to provision AWS resources (bucket, IAM) and possibly Prefect blocks that the API can reference, but you’ll need a small post-apply step (e.g., a script) to call Prefect’s API to wire the work pool storage config. - If you’d like first-class Terraform support for work pool storage, consider opening/👍 an issue at https://github.com/PrefectHQ/terraform-provider-prefect/issues and we can track it. If you tell me your desired bucket/credential block naming scheme, I can hand you a complete, ready-to-run script you can invoke from Terraform as a null_resource/local-exec step.