Short answer: yes — this is a first‑class pattern in Prefect 3.x. You can create separate deployments that each run in their own container image and on their own work pool. A thin “orchestrator” deployment can trigger ML and PySpark deployments that run in heavier images. Your Buildkite pipeline can build/push images to ECR and then run deployment steps that reference those images. Recommended architecture - Work pools: - EKS: a Kubernetes work pool (pull-based) with a Kubernetes worker running in your EKS cluster - Optional: an ECS work pool (push-based) if you want to run some workloads in ECS/Fargate - Deployments: - Orchestrator flow: runs in a lightweight image - ML training flow: runs in a heavy ML image - PySpark flow: runs in a Spark/data image - Orchestration: The orchestrator triggers the other deployments using

run_deployment(...)

. Each downstream deployment has its own image so they pull exactly the environment they need. Two good ways to wire source code + images - Source pulled at runtime (recommended for decoupling): - Each deployment points to code via

flow.from_source(...)

- Each deployment sets the

image

to the ECR image that has the right runtime deps (no build/push from Prefect) - Code baked into image: - CI builds images that include your flow code - Deployments reference those prebuilt images (

build=False

,

push=False

) Example: three flows, three images, run on EKS

python
# flows/orchestrator.py
from prefect import flow
from prefect.deployments import run_deployment

@flow
def orchestrate():
    # Wait for ML training to finish
    ml_run = run_deployment(
        name="ml-train/prod",
        parameters={"epochs": 5},
        timeout=3600
    )
    # Fire-and-forget PySpark job
    spark_run = run_deployment(
        name="pyspark-extract/prod",
        parameters={"date": "2025-01-01"},
        timeout=0
    )
    return {"ml_run_id": ml_run.id, "spark_run_id": spark_run.id}

python
# flows/ml_train.py
from prefect import flow

@flow
def ml_train(epochs: int = 1):
    import torch  # heavy deps live in the ML image
    return {"trained": True}

python
# flows/pyspark_job.py
from prefect import flow

@flow
def pyspark_extract(date: str):
    from pyspark.sql import SparkSession  # heavy deps live in Spark image
    spark = SparkSession.builder.getOrCreate()
    return {"date": date}

Deployment script (works well in Buildkite) - Assumes images are already built/pushed by CI: ECR URIs like

<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/...:sha-|123456789012.dkr.ecr.us-east-1.amazonaws.com/...:sha->...

- Uses

from_source

to pull code at runtime; images only provide the right environment ```python # deploy.py import os from prefect import flow from prefect.deployments import run_deployment # Orchestrator from prefect import flow as flow_decorator orchestrator = flow.from_source( source="https://github.com/your-org/orchestrator-repo.git", entrypoint="flows/orchestrator.py:orchestrate", ) orchestrator.deploy( name="prod", work_pool_name="eks-kubernetes-pool", image=os.environ["ECR_ORCH_IMAGE"], # e.g., .../orchestrator:sha-abc build=False, push=False, job_variables={ "namespace": "data-platform", "service_account_name": "prefect-worker", "image_pull_policy": "IfNotPresent", # Optional resources, env, labels, imagePullSecrets: # "image_pull_secrets": ["ecr-pull-secret"], # "env": {"PREFECT_LOGGING_LEVEL": "INFO"}, # "resources": {"requests": {"cpu": "500m", "memory": "1Gi"}, # "limits": {"cpu": "1", "memory": "2Gi"}}, }, ) # ML training ml_train = flow.from_source( source="https://github.com/your-org/ml-repo.git", entrypoint="flows/ml_train.py:ml_train", ) ml_train.deploy( name="prod", work_pool_name="eks-kubernetes-pool", image=os.environ["ECR_ML_IMAGE"], # e.g., .../ml-train:sha-def build=False, push=False, job_variables={

"namespace": "data-platform", "service_account_name": "prefect-worker", "image_pull_policy": "IfNotPresent", "resources": {"requests": {"cpu": "2", "memory": "8Gi"}, "limits": {"cpu": "4", "memory": "16Gi"}}, }, ) # PySpark pyspark = flow.from_source( source="https://github.com/your-org/spark-repo.git", entrypoint="flows/pyspark_job.py:pyspark_extract", ) pyspark.deploy( name="prod", work_pool_name="eks-kubernetes-pool", image=os.environ["ECR_SPARK_IMAGE"], # e.g., .../pyspark:sha-ghi build=False, push=False, job_variables={ "namespace": "data-platform", "service_account_name": "prefect-worker", "image_pull_policy": "IfNotPresent", # Include your Spark runtime/configs as needed }, ) ``` Buildkite outline - Step 1: Build and push images (orchestrator, ml, pyspark) to ECR with immutable SHA tags - Step 2: Deploy Prefect flows by running

python deploy.py

with env vars set: -

ECR_ORCH_IMAGE

,

ECR_ML_IMAGE

,

ECR_SPARK_IMAGE

- Step 3: (Once) Create work pool and install a Kubernetes worker in EKS Notes for EKS + ECR pulls - Ensure your EKS nodes (or Fargate profile) have permissions to pull from your ECR registry, or configure

image_pull_secrets

in

job_variables

. - Common k8s

job_variables

for the Kubernetes pool:

namespace

,

service_account_name

,

image_pull_policy

,

env

,

labels

,

annotations

,

image_pull_secrets

, and resource requests/limits. ECS option (if you want some heavy compute on ECS) - Create an ECS work pool and deploy those flows with

work_pool_name="ecs-pool"

and an ECR image - Use

job_variables

to control CPU/memory/launch type, etc. - Orchestrator still uses

run_deployment(...)

to trigger ECS-backed deployments Gotchas and tips - Keep the orchestrator image slim: only Prefect + light deps. Don’t import ML or Spark packages in orchestrator code paths. - Prefer immutable image references (digests) in deployments for reproducibility. - If you need to tweak infra per run, you can pass

job_variables

to

run_deployment(...)

to override things like env vars or resources at runtime. - For large result passing, prefer durable storage (S3) rather than returning large in-memory results between deployments. Docs you’ll likely want - Work pools: Work pools concept - Workers: Workers concept - Kubernetes: Run flows on Kubernetes - AWS ECS: ECS worker (prefect-aws) - Deploy via Python: Deployments via Python - Run deployments programmatically: Run deployments (Python and CLI) - Python API for run_deployment: API: prefect.deployments.flow_runs.run_deployment - CI/CD with Prefect: CI/CD guide Open questions to tailor this further - Will ML and PySpark workloads run only on EKS, or do you also want some on ECS/Fargate? - Do you prefer code pulled from Git at runtime (images as runtimes) or code baked into each image? - Can your EKS nodes pull from all target ECR repos without imagePullSecrets, or should we set those in

job_variables

? If you share which way you want to package code (pull from Git vs baked-in), I can provide a Buildkite snippet and the exact work-pool and worker setup for EKS (and ECS if desired).

also @Marvin if you can answer these ^

I’m happy to help! I don’t see the earlier questions in this thread—could you paste them here or summarize what you’d like me to answer? If it’s about Prefect, sharing a bit of context will help me give precise guidance: - Prefect version (I’ll assume 3.x unless you say otherwise) - How you’re running flows (local, worker + work pool, Kubernetes, Docker, Cloud) - Deployment method (e.g.,

flow.from_source(...).deploy(...)

) - Any error messages/logs and a brief code snippet if relevant Drop the questions and I’ll tackle them one by one.