Short answer: yes—this is a well-supported and common pattern in Prefect on EKS. Use a lightweight “orchestration” deployment to trigger other deployments that each run in their own Docker image (ML, Spark, etc.) via Kubernetes work pools. Here’s how to structure it. High-level architecture - Prefect Server (OSS) reachable by your worker(s) and CI. - One or more Kubernetes work pools in EKS. Each flow run becomes a Kubernetes Job with a container image defined per deployment or per-run. - Separate Docker images in ECR: - Orchestration: minimal Prefect + your flow coordination code - ML training: heavy deps (GPU libs, frameworks) - Data processing: PySpark + spark-submit tooling (or Spark Operator/EMR on EKS, see notes below) - Separate deployments per concern that set their image via job variables. - The orchestration flow triggers the ML/PySpark deployments as needed. Triggering other flows from a flow - Use

run_deployment

(Prefect 3.x) to start other deployments from your orchestration flow. You can choose to wait or fire-and-forget. Example

python
from prefect import flow
from prefect.deployments import run_deployment

@flow
def orchestrate_training_and_spark(customer_id: int):
    # trigger ML training and wait for completion
    ml_run = run_deployment(
        name="ml-train/train",
        parameters={"customer_id": customer_id},
        timeout=3600,  # wait up to an hour
    )

    # trigger PySpark job and return immediately (fire-and-forget)
    spark_run = run_deployment(
        name="spark/transform",
        parameters={"customer_id": customer_id},
        timeout=0,            # do not wait
        as_subflow=True       # show as subflow in the UI (default)
    )
    return {"ml_state": ml_run.state.name, "spark_run_id": str(spark_run.id)}

How images are chosen per run on EKS - Kubernetes work pools have a base job template with variables like

image

,

image_pull_policy

, etc. - Each deployment can override the container image via

job_variables

. - You can also override per-run by passing

job_variables

to

run_deployment

if you need to pin a specific image tag dynamically. Defining deployments with different images - In Python:

python
from prefect import flow

@flow
def train(...): ...
@flow
def transform(...): ...

# Deploy with image overrides
train.from_source(".").deploy(
    name="train",
    work_pool_name="eks-default",
    job_variables={"image": "<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/ml:sha-abc123|123456789012.dkr.ecr.us-east-1.amazonaws.com/ml:sha-abc123>"}
)

transform.from_source(".").deploy(
    name="transform",
    work_pool_name="eks-default",
    job_variables={"image": "<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/spark:sha-def456|123456789012.dkr.ecr.us-east-1.amazonaws.com/spark:sha-def456>"}
)

- Or in prefect.yaml (recommended if your CI builds images and then calls

prefect deploy

). You can wire CI outputs (git SHA tags, image URIs) into job_variables. Kubernetes setup on EKS - Create a Kubernetes work pool:

prefect work-pool create "eks-default" --type kubernetes

- Start a Kubernetes worker inside your EKS cluster (recommended). It needs K8s API access and Prefect API access:

prefect worker start -p eks-default

- Install the Kubernetes extra in that worker environment:

pip install "prefect[kubernetes]"

- Auth to ECR: - Typically handled by node IAM roles in EKS; you may not need imagePullSecrets if nodes can pull from your ECR registry. - Cross-account or more restrictive setups may require

imagePullSecrets

. Buildkite CI/CD - Build and push three images (orchestration, ml, spark) to ECR (tag with git SHA). - Deploy your Prefect flows referencing those images: - Option A: Have prefect.yaml in each repo and run

prefect deploy

. - Option B: Use Python API in a CI step to call

.deploy(...)

with

job_variables={"image": "<ECR_URI>:<SHA>"}

. - You can maintain separate repos for orchestration, ML, and Spark flows or a monorepo with separate Dockerfiles and deployments. Patterns and options for Spark

- Simple: Put spark + spark-submit in your “data processing” image. The Prefect flow run can execute spark-submit in-cluster (the driver runs in the same pod; executors in separate pods). - Operator: If using the Spark-on-Kubernetes Operator (SparkApplication CRDs), you can: - Launch spark-submit from the flow run container to create SparkApplication objects, or - Use a small adapter in your flow to

kubectl apply

or call the K8s API for the CRD. - EMR on EKS: Orchestrate EMR-on-EKS jobs from Prefect via boto3. The Prefect flow run is just the control plane, and EMR handles Spark infra. Operational tips - Separate pools per workload class can help with routing and cluster policies: - eks-ml (GPU node selectors/tolerations, service account with S3/ECR access) - eks-spark (separate namespace, higher memory) - eks-orchestration (lightweight defaults) - Use

as_subflow=True/False

depending on whether you want the downstream run to appear nested in the UI. - Decide whether the orchestration flow should wait for downstream runs: - Wait (

timeout

> 0) when you need results or gating. - Fire-and-forget (

timeout=0

) for async pipelines and fan-out. - For Spark: Prefect will capture the flow container logs. Spark executor logs are separate; configure cluster logging (e.g., CloudWatch/Fluent Bit) if you need centralized Spark logs. - Make sure PREFECT_API_URL is set for all worker pods and they can reach your OSS server. - If you need per-run tweaks (e.g., CPU/memory), add more job_variables or bake them into the work pool’s base job template. Docs and references - Trigger deployments from flows: Run deployments from flows - Run flows on Kubernetes (EKS): Run flows on Kubernetes - Docker integration and image build steps: prefect-docker integration A couple of questions to help tailor specifics: - Do you want the orchestration flow to wait for ML/Spark completion or trigger them asynchronously? - For Spark, are you planning spark-submit in a container, the Spark Operator (CRDs), or EMR on EKS? - Any GPU requirements for ML? If so, we can show how to set node selectors/tolerations and resource requests in your job template.