It sounds like Prefect is bundling your whole project directory when you run a deployment, which ends up including your SageMaker processing script. In Prefect 3.x there are a few clean ways to avoid that, depending on how you want SageMaker to get the script. Options: 1) Exclude the script with .prefectignore (keep using local project packaging) - Add a

.prefectignore

file at your project root. It uses .gitignore-style patterns. - Example:

# Exclude SageMaker processing code from the deployment artifact
sagemaker_processing.py
sagemaker_scripts/
*.ipynb

# Exclude large data, etc.
data/**
!data/README.md

This lets your Prefect deployment package stay small, while your SageMaker job can still pull the script from wherever you intend (e.g., S3, separate repo, or baked into the SageMaker image). 2) Deploy from Git and skip packaging entirely - Use

Flow.from_source(...).deploy(...)

with

build=False

and

push=False

. This prevents Prefect from packaging any local files; the worker will pull your code from Git at run time. - Example:

from prefect import flow

@flow
def my_flow():
    # Your flow that kicks off the SageMaker processing job
    ...

if __name__ == "__main__":
    flow.from_source(
        source="<https://github.com/your-org/your-repo.git@main>",
        entrypoint="flows/sm_flow.py:my_flow",
    ).deploy(
        name="sm-ecs",
        work_pool_name="your-ecs-ec2-pool",
        build=False,
        push=False,
    )

You can keep the SageMaker processing script in a separate repo (or path) that SageMaker pulls from, while the Prefect flow repo remains lean. If you prefer the CLI, the same idea applies: configure your

prefect.yaml

to pull from Git and skip build/push. For example:

# prefect.yaml
deployments:
- name: sm-ecs
  entrypoint: flows/sm_flow.py:my_flow
  work_pool:
    name: your-ecs-ec2-pool
  build: false
  push: false
  pull:
    - prefect.deployments.steps.git_clone:
        repository: <https://github.com/your-org/your-repo.git>
        branch: main

3) Put the processing script where SageMaker expects it, independently of Prefect Common patterns: - Store the script in S3 and reference it as a

ProcessingInput

for the job. - Bake the script into the ECR image used by the SageMaker Processing job (keeps Prefect completely out of that artifact path). Given your setup (ECS EC2 worker, single EC2 host for all envs), the simplest fix is usually: - Add

.prefectignore

to exclude the SageMaker script from Prefect’s package, and - Ensure the SageMaker job fetches the script from S3 or an image. Quick questions to tailor this: - Are you deploying via

prefect deploy

(CLI) or

flow.from_source(...).deploy()

in code? - Where do you want SageMaker to fetch the processing script from (S3, ECR image, or a separate repo)?

Great — that approach works well. Here’s a clean pattern to implement it with a Prefect YAML deployment: Plan - Exclude the SageMaker script from the Prefect build with a .prefectignore so it’s never bundled into the deployment artifact. - In CI/CD: upload the script to S3 and set a deployment env/variable to use the S3 path. - In local dev: point to a local file path and have your flow use that instead. - Use two deployments (dev/prod) or Prefect Variables to switch the source at runtime. 1) Exclude the script from Prefect’s package Add a .prefectignore at your project root:

# Do not include the SageMaker processing code in the Prefect deployment artifact
sagemaker_scripts/**
sagemaker_processing.py

# Optional: exclude other large/local files
data/**
!data/README.md

2) Prefect YAML with two deployments (dev uses local; prod uses S3)

# prefect.yaml
name: sagemaker-example
prefect-version: ">=3.0.0"

deployments:
- name: sm-ecs-dev
  entrypoint: flows/sm_flow.py:my_flow
  work_pool:
    name: your-ecs-ec2-pool
  build: true
  push: true
  environment:
    SAGEMAKER_SCRIPT_MODE: local
    SAGEMAKER_SCRIPT_LOCAL_PATH: sagemaker_scripts/processing.py
    # S3 not used in dev
    SAGEMAKER_SCRIPT_S3_URI: ""

- name: sm-ecs-prod
  entrypoint: flows/sm_flow.py:my_flow
  work_pool:
    name: your-ecs-ec2-pool
  build: true
  push: true
  environment:
    SAGEMAKER_SCRIPT_MODE: s3
    SAGEMAKER_SCRIPT_LOCAL_PATH: ""
    SAGEMAKER_SCRIPT_S3_URI: <s3://my-bucket/sagemaker/processing/processing.py>

Notes: - You can also keep a single deployment and use Prefect Variables instead of environment if you prefer; the switching logic is the same. - Since you’re on ECS EC2, the “local path” only works for dev runs that execute outside of ECS or if you mount the file into the container. Most teams keep dev runs local (Process worker) and prod in ECS. 3) Flow code that switches between local and S3 Below is a minimal pattern. For local dev, point ScriptProcessor to a local file (the SageMaker SDK will upload it for that run). For prod, point to S3 and run the script in your container (either via Processor with command or by downloading the S3 object first).

import os
from prefect import flow, get_run_logger

@flow
def my_flow():
    logger = get_run_logger()

    mode = os.getenv("SAGEMAKER_SCRIPT_MODE", "local")
    local_path = os.getenv("SAGEMAKER_SCRIPT_LOCAL_PATH", "sagemaker_scripts/processing.py")
    s3_uri = os.getenv("SAGEMAKER_SCRIPT_S3_URI", "")

    if mode == "local":
        <http://logger.info|logger.info>(f"Using local script: {local_path}")
        # Example: use ScriptProcessor which will package and upload your local script
        # from sagemaker.processing import ScriptProcessor
        # sp = ScriptProcessor(...your config...)
        # sp.run(code=local_path, arguments=[...], inputs=[...], outputs=[...], wait=True)
    else:
        <http://logger.info|logger.info>(f"Using S3 script: {s3_uri}")
        # Option A: Use Processor and run the S3-hosted script via command after mounting as ProcessingInput
        # from sagemaker.processing import Processor, ProcessingInput
        # proc = Processor(...your config...)
        # proc.run(
        #   inputs=[ProcessingInput(source=s3_uri, destination="/opt/ml/processing/code")],
        #   code=None,  # not using ScriptProcessor here
        #   arguments=[...],
        #   wait=True,
        #   # If needed, provide 'command' so container runs the script:
        #   # command=["python", "/opt/ml/processing/code/processing.py"]
        # )
        #
        # Option B: Download S3 to a temp path first, then still use ScriptProcessor.run(code=<local_path>)
        # (useful if you want to stick with ScriptProcessor for both modes)

4) CI/CD step to upload the script to S3 - Add a step before `prefect deploy`:

aws s3 cp sagemaker_scripts/processing.py <s3://my-bucket/sagemaker/processing/processing.py> --region ap-southeast-1

- Ensure your ECS task role can read from that S3 location at runtime. Why this works - .prefectignore keeps the SageMaker script out of the deployment artifact so Prefect doesn’t try to package or ship it with your flow. - Your flow decides where to read the script from based on env/variables. - Dev stays fast and simple (local file). Prod is deterministic (S3). If you want, I can help tailor the YAML to your exact project structure or suggest the cleanest SageMaker pattern (ScriptProcessor vs Processor) based on the container you’re using.