Madushan Pathirana
08/11/2025, 5:54 PMMarvin
08/11/2025, 5:54 PMMarvin
08/11/2025, 5:59 PMpip install "prefect[docker]"
1) Prebuild your image and push to ECR, then deploy from Python
- Create a Docker work pool (one-time):
prefect work-pool create my-docker-pool --type docker
- Build and push your image to ECR (one-time per version/tag):
- Authenticate Docker to ECR:
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <http://123456789012.dkr.ecr.us-east-1.amazonaws.com|123456789012.dkr.ecr.us-east-1.amazonaws.com>
- Build and push:
docker build -t <http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest> .
docker push <http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest>
- Your flow code (pure Python deploy, no YAML):
from prefect import flow, task
import httpx
@task(retries=2)
def get_cat_fact():
url = "<https://catfact.ninja/fact>"
with httpx.Client() as client:
response = client.get(url)
response.raise_for_status()
fact = response.json()["fact"]
print(f"Cat Fact: {fact}")
return fact
@flow(name="Cat Fact Flow", log_prints=True)
def cat_fact_flow():
print("Starting the cat fact flow!")
for _ in range(3):
get_cat_fact()
print("Flow finished successfully.")
if __name__ == "__main__":
cat_fact_flow.deploy(
name="my-deployment",
work_pool_name="my-docker-pool",
image="<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest>",
build=False, # image already built
push=False, # image already pushed
)
- Start a worker to pick up runs:
prefect worker start -p my-docker-pool
Notes:
- Make sure your image includes your dependencies (e.g., prefect
, httpx
, and any others your flow needs). If you’re not copying your code into the image, point your deployment at a repo instead (see next section).
- If your worker runs on another machine, it must have permission to pull from ECR (IAM creds or a running docker login
).
Optional: pull your code at runtime from a repo (no code baked into the image)
- If you don’t want to bake your flow code into the image, you can deploy from a git repo; the container just needs Prefect (and git if required) installed:
cat_fact_flow.from_source(
source="<https://github.com/you/your-repo>",
entrypoint="path/to/script.py:cat_fact_flow",
).deploy(
name="my-deployment",
work_pool_name="my-docker-pool",
image="<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest>",
build=False,
push=False,
)
2) Let Prefect build and push the image for you (also from Python)
If you’d rather Prefect handle the build/push (still no YAML), you can pass a Docker image object. You’ll need to be logged into ECR beforehand (same aws ecr get-login-password | docker login
as above).
```
from prefect import flow, task
from prefect.docker import DockerImage
import httpx
@task(retries=2)
def get_cat_fact():
url = "https://catfact.ninja/fact"
with httpx.Client() as client:
response = client.get(url)
response.raise_for_status()
print(response.json()["fact"])
@flow(name="Cat Fact Flow", log_prints=True)
def cat_fact_flow():
for _ in range(3):
get_cat_fact()
if name == "__main__":
cat_fact_flow.deploy(
name="my-deployment",
work_pool_name="my-docker-pool",
image=DockerImage(
name="123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat",
tag="v1",
dockerfile="auto",
# Additional build kwargs may be supported; ensure your deps end up in the image.Marvin
08/11/2025, 5:59 PMimage
in code should be a plain string; don’t use Slack link formatting inside code.
- If you use the prebuilt image path, set build=False, push=False
.
- Ensure the image has all runtime deps (prefect, httpx, git if cloning, any auth libs).
- The worker host must be able to pull from your ECR registry.
If you share how you prefer to include your code (baked into the image vs pulled from a repo) and your AWS region/account ID, I can give you a tailored Dockerfile or the exact deploy snippet.Madushan Pathirana
08/12/2025, 2:49 AMMarvin
08/12/2025, 2:49 AMMarvin
08/12/2025, 2:49 AMprefect work-pool create my-docker-pool --type docker
2) Make sure you can push to ECR
- Create the repo if needed:
aws ecr create-repository --repository-name prefect-cat --region us-east-1
- Log in Docker to ECR:
aws ecr get-login-password --region us-east-1 \
| docker login --username AWS --password-stdin <http://123456789012.dkr.ecr.us-east-1.amazonaws.com|123456789012.dkr.ecr.us-east-1.amazonaws.com>
3) Deploy from Python using a local Dockerfile and push to ECR
- Put your Dockerfile next to your script (or provide a path to it).
- Prefect will use your local filesystem to build the image, then push to ECR because the image name points there.
Example:
from prefect import flow, task
from prefect.docker import DockerImage
import httpx
@task(retries=2)
def get_cat_fact():
with httpx.Client() as client:
r = client.get("<https://catfact.ninja/fact>")
r.raise_for_status()
print(r.json()["fact"])
@flow(name="Cat Fact Flow", log_prints=True)
def cat_fact_flow():
for _ in range(3):
get_cat_fact()
if __name__ == "__main__":
cat_fact_flow.deploy(
name="my-deployment",
work_pool_name="my-docker-pool",
image=DockerImage(
name="<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat>",
tag="v1",
dockerfile="Dockerfile", # local file; use path if it's elsewhere
),
build=True, # build locally using the Dockerfile
push=True, # push to ECR (must be logged in)
)
Notes:
- The registry is inferred from the image name. Point it at ECR and Prefect will push there.
- Build context is your current working directory when you run the script; make sure the Dockerfile and any referenced files are within it.
- Alternatively, prebuild and push your image yourself, then deploy with build=False, push=False
and image="123...ecr.../repo:tag"
.
If you want me to sanity-check your Dockerfile or image name, paste them here and I’ll spare you the trial-and-error.Madushan Pathirana
08/12/2025, 2:52 AMMarvin
08/12/2025, 2:52 AMMarvin
08/12/2025, 2:53 AMbuild=False, push=False
, then yes, you must push it to ECR at least once yourself.
- If you let Prefect build and push, then no pre-push is needed. Just make sure:
- You’re logged into ECR (aws ecr get-login-password ... | docker login ...
)
- The ECR repository already exists (Prefect won’t create it)
Example (Prefect does the first push):
from prefect import flow, task
from prefect.docker import DockerImage
@task
def t(): print("hi")
@flow
def f(): t()
if __name__ == "__main__":
f.deploy(
name="ecr-deploy",
work_pool_name="my-docker-pool",
image=DockerImage(
name="<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat>",
tag="v1",
dockerfile="Dockerfile", # local Dockerfile
),
build=True, # build locally
push=True, # push to ECR (must be logged in)
)
If the repo doesn’t exist yet:
aws ecr create-repository --repository-name prefect-cat --region us-east-1
And don’t forget: your worker must be able to pull from ECR (credentials/permissions on the worker host).Madushan Pathirana
08/12/2025, 5:12 AMMarvin
08/12/2025, 5:12 AMMarvin
08/12/2025, 5:13 AMpush: false
from job_variables
(that’s not a Docker worker job var) and wired the push step to the build step’s outputs.
name: src
prefect-version: 3.4.11
build:
- prefect_docker.deployments.steps.build_docker_image:
id: build_image
requires: prefect-docker>=0.3.1
image_name: <http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-flows|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-flows>
tag: latest
dockerfile: Dockerfile.prefect_flows
push:
- prefect_docker.deployments.steps.push_docker_image:
requires: prefect-docker>=0.3.1
image_name: '{{ build_image.image_name }}'
tag: '{{ build_image.tag }}'
pull:
- prefect.deployments.steps.set_working_directory:
directory: /opt/prefect/src
deployments:
- name: local-test
entrypoint: flows/test_flow.py:cat_fact_flow
parameters: {}
schedule: {}
tags: []
work_pool:
name: my-docker-pool-2
work_queue_name: default
job_variables:
image: '{{ build_image.image }}' # resolves to "<image_name>:<tag>"
What you need to do before running `prefect deploy`:
- Create the ECR repo once (if it doesn’t exist):
aws ecr create-repository --repository-name prefect-flows --region us-east-1
- Log in Docker to ECR (where you run prefect deploy
):
aws ecr get-login-password --region us-east-1 \
| docker login --username AWS --password-stdin <http://123456789012.dkr.ecr.us-east-1.amazonaws.com|123456789012.dkr.ecr.us-east-1.amazonaws.com>
- Ensure the environment has prefect-docker>=0.3.1
installed.
Then:
prefect deploy
prefect worker start -p my-docker-pool-2
That will:
- Build your image locally using Dockerfile.prefect_flows
- Tag it as <http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-flows:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-flows:latest>
- Push it to ECR (because of the push step)
- Register the deployment pointing its image
to the built tag
Finally, your worker must be able to pull from ECR (IAM creds or a prior docker login
on the worker host). Otherwise it will sulk and refuse to run your flow, a bit like me on Mondays.