Short answer: yes—there are ways to cut down S3 downloads, but it depends on how you’re pulling code and how your workers run. In Prefect 3.x, the built-in S3 “remote storage” pull does not do change detection; it will re-download on each pull. Here are practical options: What’s happening today (likely) - If you’re using S3 via

flow.from_source(... RemoteStorage ...)

or the

pull_from_remote_storage

deployment step, Prefect pulls files on a schedule (default every 60s) and before runs. There’s no hash/ETag comparison—S3 content is re-downloaded when a pull occurs. Ways to reduce/avoid repeated downloads 1) Increase or disable S3 pull frequency - Set a larger

pull_interval

(seconds) or disable it with

None

to avoid per-run pulls. - Example:

from prefect import flow
  from prefect.runner.storage import RemoteStorage

  my_flow = flow.from_source(
      source=RemoteStorage(
          url="<s3://my-bucket/flows/code>",
          pull_interval=3600  # re-pull at most once an hour
          # or pull_interval=None to skip periodic pulls entirely
      ),
      entrypoint="flows.py:my_flow",
  )

- Important: If you set

pull_interval=None

, make sure your worker is long-lived (so it pulls once at startup and reuses the local cache). If you spin up a fresh, ephemeral worker per run (e.g., a new K8s Job), there won’t be a prior cache and you’ll still incur a download. 2) Version your S3 path (simple and effective) - Upload code under a versioned prefix (e.g., `s3://bucket/flows/build-<gitsha>/...`) and point your deployment to that exact path. Bump the path only when you push changes.

source=RemoteStorage(
      url="<s3://my-bucket/flows/build-abc123>",
      pull_interval=None
  )

- This gives you “download once per build,” not per run. 3) Prefer Git for code (change-aware) - If feasible, store code in Git and use

GitRepository

with a

commit_sha

. Prefect will skip re-pulls when the commit hasn’t changed:

from prefect.runner.storage import GitRepository

  source=GitRepository(
      url="<https://github.com/org/repo.git>",
      commit_sha="abc123def456",
      pull_interval=None  # no periodic pulls
  )

- This is the most bandwidth-friendly approach for code updates. 4) Persist a local cache on the worker - If you’re on Kubernetes or Docker, use a long-lived worker and mount a persistent volume for its working directory. That way the pulled code stays on disk between runs and only refreshes when your pull policy triggers. 5) Bake code into the image - For fully stable code that changes infrequently, build it into your container image and run from that image. Then you only “download” the code when pulling a new image version, not per flow run. Notes and caveats - Remote S3 pulls don’t do incremental sync by file hash today; it’s a full copy when a pull happens. - If your infra creates a brand-new worker per run, set a longer

pull_interval

won’t help—use versioned S3 paths, Git with commit SHAs, or baked images. - If you must stick with S3 and ephemeral workers, versioned S3 prefixes are the most reliable “only when changed” trigger. A couple of quick questions so I can tailor the exact steps: - Are you using

flow.from_source(... RemoteStorage ...)

or the

pull_from_remote_storage

step in deployments? - What work pool/worker type are you using (Docker, Kubernetes, Process), and are your workers long-lived or ephemeral per run? - Could you switch to versioned S3 prefixes or to Git for code while keeping data in S3?

Thanks — that helps. The short version:

prefect_aws.deployments.steps.pull_from_s3

always re-downloads every file each time it runs. It doesn’t have caching, change detection, or a pull interval, so you’ll see repeated downloads on both Docker and Process workers unless you change the approach. Here are proven ways to speed things up, tailored to each worker type. What you’re using today - `prefect_aws.deployments.steps.pull_from_s3`: - Uses boto3 and downloads every object under the folder on each execution. - No incremental sync, no skip-if-unchanged behavior. Options for Docker workers - Best: Bake the code into the image (no runtime download)

build:
    - prefect_docker.deployments.steps.build_docker_image:
        image_name: my-registry/prefect-flows
        tag: v1
        dockerfile: auto

  push:
    - prefect_docker.deployments.steps.push_docker_image:
        image_name: my-registry/prefect-flows
        tag: v1

  deployments:
    - name: my-deployment
      entrypoint: flows.py:my_flow
      work_pool:
        name: my-docker-pool
        job_variables:
          image: my-registry/prefect-flows:v1

- Rebuild/push the image only when code changes; at runtime there’s nothing to download. - If you must fetch from S3 at runtime: use awscli sync (incremental) instead of

pull_from_s3

pull:
    - prefect.deployments.steps.set_working_directory:
        directory: /opt/prefect/code
    - prefect.deployments.steps.run_shell_script:
        script: |
          aws s3 sync <s3://my-bucket/my-prefix> . --no-progress --only-show-errors

- This only transfers changed files. - Ensure your image includes AWS CLI credentials + permissions. - Or mount a persistent host volume as a cache across runs - Configure your Docker work pool’s job variables to mount a host path into the container (so repeated runs on the same host reuse files). Example:

work_pool:
      name: my-docker-pool
      job_variables:
        image: my-registry/prefect-flows:base
        volumes:
          - /var/prefect/code-cache:/opt/prefect/code

- Then use either

aws s3 sync

or your current step once per host; subsequent runs hit the cache. Options for Process workers - Prefer Git with commit pinning (change-aware) - Swap S3 code pulls for:

pull:
      - prefect.deployments.steps.git_clone:
          repository: <https://github.com/org/repo.git>
          commit: abc123def456   # pin exact commit to avoid re-pulls if already current

- On a long-lived Process worker, the checkout will be reused between runs and only update when the commit changes. - If you must stay on S3, use incremental sync instead of

pull_from_s3

pull:
    - prefect.deployments.steps.set_working_directory:
        directory: /opt/prefect/code
    - prefect.deployments.steps.run_shell_script:
        script: |
          aws s3 sync <s3://my-bucket/my-prefix> . --no-progress --only-show-errors

- On a long-lived Process worker (same machine), this will download once then only deltas. Cross-cutting improvements - Version your S3 prefix - Upload code to

<s3://bucket/flows/build-<gitsha>/>...

and point deployments at that exact prefix. You only “change” the prefix when the code changes. - Combine with Docker image baking or incremental sync for predictable updates. - Use

flow.from_source(...)

with a long-lived worker - If you switch to

flow.from_source(source="<s3://bucket/prefix>", entrypoint=...)

, you can set a longer

pull_interval

on RemoteStorage to avoid frequent pulls on long-lived Process workers. Note: for Docker (ephemeral containers), a longer interval won’t help because each run starts fresh. If you share: - Whether you can bake code into your Docker image, - Whether you can install/use AWS CLI in your environments, - If Git is an option for code while keeping data in S3, …I can provide an exact

prefect.yaml

or code snippet tailored to your setup.

Great question. With

prefect_aws.deployments.steps.pull_from_s3

, files are downloaded to whatever the current working directory (cwd) is when that step runs. - If you include

prefect.deployments.steps.set_working_directory

before

pull_from_s3

, that’s where the files will go. - If you don’t set it, the cwd depends on the worker type. Where they land by worker type - Process worker - Default: a per-run temporary directory (OS temp). It’s cleaned up after the run, so nothing persists by default. - If you configure a persistent working directory for the worker (or set it via a pull step), files will be saved there and will persist on that host. - Example:

pull:
      - prefect.deployments.steps.set_working_directory:
          directory: /srv/prefect/flows
      - prefect_aws.deployments.steps.pull_from_s3:
          bucket: my-bucket
          folder: my-project

- Docker worker - Default container cwd: /opt/prefect (the Prefect image WORKDIR). Files go there. - They only persist across runs if you mount a host volume to that path and reuse it across runs. - Example (job variables on the work pool):

work_pool:
      name: my-docker-pool
      job_variables:
        volumes:
          - /var/prefect/code-cache:/opt/prefect

Then

pull_from_s3

will write into /opt/prefect in the container, which is /var/prefect/code-cache on the host — reusable across runs on that node. Quick way to see where it’s writing - Add a tiny step before/after the S3 pull:

- prefect.deployments.steps.run_shell_script:
      script: |
        echo "CWD is: $(pwd)"
        ls -la

Key takeaway: by default, files do not persist (temp dir for Process; ephemeral container FS for Docker). They only persist if you explicitly point the cwd to a persistent path (Process) or mount a persistent volume at the container’s workdir (Docker).