Fugo Takefusa
09/18/2025, 10:52 PMMarvin
09/18/2025, 10:52 PMMarvin
09/18/2025, 10:56 PM- name: Build and push
run: |
docker build -t $ECR_URI:$GIT_SHA .
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_URI
docker push $ECR_URI:$GIT_SHA
3) Create/run an ECS work pool and worker (once)
- Create a work pool in Prefect and start a worker that targets your ECS cluster; the worker needs AWS IAM permissions for ECS (and ECR if pulling private images).
- Recommended to run the worker as its own tiny Fargate service so it auto-recovers and scales horizontally if needed.
Docs:
- ECS worker guide
4) Publish/update deployments from CI
- Use your repo as the source and register deployments against your ECS work pool.
- Point jobs to the ECR image you just built. In Prefect 3.x, use flow.from_source(...).deploy(...)
(deployment build methods from 2.x were removed).
Example Python-based deployment registration (works great in CI):
# deploy_register.py
import os
from prefect import flow
@flow
def etl():
pass # your flow
if __name__ == "__main__":
git_sha = os.environ["GIT_SHA"]
ecr_uri = os.environ["ECR_URI"]
work_pool = os.environ.get("WORK_POOL", "ecs-prod")
flow.from_source(
source="<https://github.com/your-org/your-repo>",
entrypoint="flows/etl.py:etl",
).deploy(
name="etl-prod",
work_pool_name=work_pool,
version=git_sha,
parameters={},
job_variables={
# ECS worker honors these; see ECS guide for available fields
"image": f"{ecr_uri}:{git_sha}",
# examples you might set:
# "cpu": 1024,
# "memory": 2048,
# "task_role_arn": "...",
# "execution_role_arn": "...",
# "env": {"ENV": "prod"},
},
)
CI step to register:
- name: Register Prefect deployment
env:
PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
GIT_SHA: ${{ github.sha }}
ECR_URI: ${{ secrets.ECR_URI }}
WORK_POOL: ecs-prod
run: python deploy_register.py
Docs:
- Store flow code and deploy from source
5) Secrets and configMarvin
09/18/2025, 10:56 PMjob_variables
or via task definitions
6) Environment separation and promotion
- Separate work pools per environment (e.g., ecs-dev
, ecs-stg
, ecs-prod
) and optionally separate VPCs/accounts
- Promote by re-registering the deployment targeting another work pool and/or by retagging the image
- Leverage deployment versioning to roll back if necessary
- Blog: Deployment versioning and rollbacks
7) Prefect Server upgrades on ECS
- Back up the database (RDS snapshot)
- Blue/green or rolling update the ECS service with a newer Prefect Server image version
- Keep your workers and deployment clients compatible with the server version (align major versions)
- Validate migrations/compatibility in a staging environment before prod
Helpful references
- ECS worker guide
- Store flow code / deploy from source
- CI/CD guidance (blog)
- Orchestrating with Docker (blog)
- Serverless with Prefect + Pulumi on AWS (blog)
Notes on CLI usage
- Common commands you’ll use include prefect work-pool create
, prefect worker start
, and prefect deploy
. I attempted to verify CLI help here but couldn’t run the checks in this environment. If you need exact flags, I’m happy to provide them and link the corresponding docs.
Questions to tailor this for you
- Which CI system are you using (GitHub Actions, GitLab CI, CircleCI, AWS CodeBuild)?
- Are you planning to run the Prefect worker as a Fargate service or elsewhere?
- Do you need multi-env promotion (dev/stage/prod) and/or multi-account AWS support?
- Any special ECS constraints (custom task defs, VPC endpoints, private subnets only)?