Ed Hartley
12/18/2025, 9:10 AMMarvin
12/18/2025, 9:10 AMMarvin
12/18/2025, 9:22 AM# prefect.yaml at repo root
pull:
# If your deployment already uses from_source (Git), Prefect will add a git pull step for you.
# Add this to install deps in the worker's environment before running the flow:
- prefect.deployments.steps.utility.pip_install_requirements:
requirements_file: requirements.txt
deployments:
- name: my-flow
entrypoint: flows/my_flow.py:my_flow # adjust to your entrypoint
work_pool:
name: default # your Process work pool
Notes:
- Make sure the worker’s Python matches what your project expects.
- This is simple but less reproducible and slower on cold starts since it installs on each run unless cached.
Option B (recommended for 50+ flows): containerize and bake dependencies into images
- Create a Docker or Kubernetes work pool and run a worker for it.
- Build images per repo (or per tech stack), push to a registry, and reference those images in your deployments.
- This gives fast startups, deterministic environments, and easy scaling across many flows.
Step-by-step:
1) Create a Docker (or Kubernetes) work pool
- In Prefect Cloud UI: Work Pools -> New -> choose Docker or Kubernetes.
- Start a worker that points to that pool:
prefect worker start -p <your-pool-name>
2) Add a prefect.yaml with image build/push and deployment wiring
- Use the prefect-docker steps to auto-build from your repo and push to your registry.
- Pin versions (ideally tag with the Git SHA) for reproducibility.
Example prefect.yaml:
# Requires: pip install prefect-docker
# Or include it in your project requirements
build:
# Optional: capture a short Git SHA for tagging
- prefect.deployments.steps.run_shell_script:
id: git-sha
script: git rev-parse --short HEAD
stream_output: false
- prefect_docker.deployments.steps.build_docker_image:
id: build-image
image_name: my-registry/my-team/my-service
tag: "{{ git-sha.stdout }}"
dockerfile: auto # auto uses prefect's generated Dockerfile
# If using requirements.txt in repo root, auto will install it.
push:
- prefect_docker.deployments.steps.push_docker_image:
image_name: "{{ build-image.image_name }}"
tag: "{{ build-image.tag }}"
deployments:
- name: my-flow
entrypoint: flows/my_flow.py:my_flow
work_pool:
name: my-docker-pool
job_variables:
image: "{{ build-image.image_name }}:{{ build-image.tag }}"
Then deploy from CI (recommended) or locally:
prefect deploy --all
3) CI/CD for many flows across many repos
- Put a prefect.yaml in each repo.
- On changes to a repo, your CI:
- logs into your registry,
- runs prefect deploy --all,
- pushes a tagged image,
- updates deployments to reference the new image tag.
- Use a small set of shared base images (e.g., one per Python version or per stack) to speed builds and centralize base dependencies.
- Tag images with the Git SHA so you can correlate a run to the exact code and deps.
When to use each:
- Option A is fine for short-lived or development flows on a Process pool.Marvin
12/18/2025, 9:22 AMflow.from_source(...).deploy(...) vs prefect deploy)
I can tailor exact config and CI examples for your setup and help convert one repo end-to-end.