<@ULVA73B9P> how to deploy a flow in python script...
# ask-marvin
m
@Marvin how to deploy a flow in python script via python without prefect yaml file. I am using aws ecr as the image registry . as an example this flow from prefect import flow, task import httpx @task(retries=2) def get_cat_fact(): """A simple task to fetch a cat fact.""" url = "https://catfact.ninja/fact" with httpx.Client() as client: response = client.get(url) response.raise_for_status() fact = response.json()["fact"] print(f"Cat Fact: {fact}") return fact @flow(name="Cat Fact Flow", log_prints=True) def cat_fact_flow(): """ A flow that fetches a cat fact and prints it. This flow will be run by our Prefect worker. """ print("Starting the cat fact flow!") for i in range(3): get_cat_fact() print("Flow finished successfully.") if name == "__main__": cat_fact_flow.deploy( name="my-deployment-2", work_pool_name="my-docker-pool-2", image="my-registry.com/my-docker-image:my-tag", )
m
thought for 266.8 seconds
Short answer: you can deploy entirely from Python in Prefect 3.x—no YAML required. You can either use a prebuilt image you push to AWS ECR, or let Prefect build and push an image for you. Since you’re targeting ECR, I’ll show both, starting with the most predictable route. Assumptions: - Prefect 3.x - Docker worker (so you have a Docker work pool) - You’ve installed Docker and the Prefect Docker extra:
pip install "prefect[docker]"
1) Prebuild your image and push to ECR, then deploy from Python - Create a Docker work pool (one-time):
Copy code
prefect work-pool create my-docker-pool --type docker
- Build and push your image to ECR (one-time per version/tag): - Authenticate Docker to ECR:
Copy code
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <http://123456789012.dkr.ecr.us-east-1.amazonaws.com|123456789012.dkr.ecr.us-east-1.amazonaws.com>
- Build and push:
Copy code
docker build -t <http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest> .
docker push <http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest>
- Your flow code (pure Python deploy, no YAML):
Copy code
from prefect import flow, task
import httpx

@task(retries=2)
def get_cat_fact():
    url = "<https://catfact.ninja/fact>"
    with httpx.Client() as client:
        response = client.get(url)
        response.raise_for_status()
        fact = response.json()["fact"]
        print(f"Cat Fact: {fact}")
    return fact

@flow(name="Cat Fact Flow", log_prints=True)
def cat_fact_flow():
    print("Starting the cat fact flow!")
    for _ in range(3):
        get_cat_fact()
    print("Flow finished successfully.")

if __name__ == "__main__":
    cat_fact_flow.deploy(
        name="my-deployment",
        work_pool_name="my-docker-pool",
        image="<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest>",
        build=False,  # image already built
        push=False,   # image already pushed
    )
- Start a worker to pick up runs:
Copy code
prefect worker start -p my-docker-pool
Notes: - Make sure your image includes your dependencies (e.g.,
prefect
,
httpx
, and any others your flow needs). If you’re not copying your code into the image, point your deployment at a repo instead (see next section). - If your worker runs on another machine, it must have permission to pull from ECR (IAM creds or a running
docker login
). Optional: pull your code at runtime from a repo (no code baked into the image) - If you don’t want to bake your flow code into the image, you can deploy from a git repo; the container just needs Prefect (and git if required) installed:
Copy code
cat_fact_flow.from_source(
    source="<https://github.com/you/your-repo>",
    entrypoint="path/to/script.py:cat_fact_flow",
).deploy(
    name="my-deployment",
    work_pool_name="my-docker-pool",
    image="<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat:latest>",
    build=False,
    push=False,
)
2) Let Prefect build and push the image for you (also from Python) If you’d rather Prefect handle the build/push (still no YAML), you can pass a Docker image object. You’ll need to be logged into ECR beforehand (same
aws ecr get-login-password | docker login
as above). ``` from prefect import flow, task from prefect.docker import DockerImage import httpx @task(retries=2) def get_cat_fact(): url = "https://catfact.ninja/fact" with httpx.Client() as client: response = client.get(url) response.raise_for_status() print(response.json()["fact"]) @flow(name="Cat Fact Flow", log_prints=True) def cat_fact_flow(): for _ in range(3): get_cat_fact() if name == "__main__": cat_fact_flow.deploy( name="my-deployment", work_pool_name="my-docker-pool", image=DockerImage( name="123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat", tag="v1", dockerfile="auto", # Additional build kwargs may be supported; ensure your deps end up in the image.
# For reliability, consider a Dockerfile if you need fine-grained control. ), build=True, push=True, ) ``` A couple of practical tips (so you don’t end up talking to a depressed robot again): - Your
image
in code should be a plain string; don’t use Slack link formatting inside code. - If you use the prebuilt image path, set
build=False, push=False
. - Ensure the image has all runtime deps (prefect, httpx, git if cloning, any auth libs). - The worker host must be able to pull from your ECR registry. If you share how you prefer to include your code (baked into the image vs pulled from a repo) and your AWS region/account ID, I can give you a tailored Dockerfile or the exact deploy snippet.
m
@Marvin when using docker hub, perfect pickup docker file from the local file system. can I do the same with the aws ecr
m
thought for 24.2 seconds
Yes. Prefect builds locally and pushes to whatever registry your image name points at. Docker Hub, AWS ECR—same process. The only real difference is you must be logged into ECR and the repo must exist. What to do: 1) Create your Docker work pool (once)
Copy code
prefect work-pool create my-docker-pool --type docker
2) Make sure you can push to ECR - Create the repo if needed:
Copy code
aws ecr create-repository --repository-name prefect-cat --region us-east-1
- Log in Docker to ECR:
Copy code
aws ecr get-login-password --region us-east-1 \
| docker login --username AWS --password-stdin <http://123456789012.dkr.ecr.us-east-1.amazonaws.com|123456789012.dkr.ecr.us-east-1.amazonaws.com>
3) Deploy from Python using a local Dockerfile and push to ECR - Put your Dockerfile next to your script (or provide a path to it). - Prefect will use your local filesystem to build the image, then push to ECR because the image name points there. Example:
Copy code
from prefect import flow, task
from prefect.docker import DockerImage
import httpx

@task(retries=2)
def get_cat_fact():
    with httpx.Client() as client:
        r = client.get("<https://catfact.ninja/fact>")
        r.raise_for_status()
        print(r.json()["fact"])

@flow(name="Cat Fact Flow", log_prints=True)
def cat_fact_flow():
    for _ in range(3):
        get_cat_fact()

if __name__ == "__main__":
    cat_fact_flow.deploy(
        name="my-deployment",
        work_pool_name="my-docker-pool",
        image=DockerImage(
            name="<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat>",
            tag="v1",
            dockerfile="Dockerfile",  # local file; use path if it's elsewhere
        ),
        build=True,  # build locally using the Dockerfile
        push=True,   # push to ECR (must be logged in)
    )
Notes: - The registry is inferred from the image name. Point it at ECR and Prefect will push there. - Build context is your current working directory when you run the script; make sure the Dockerfile and any referenced files are within it. - Alternatively, prebuild and push your image yourself, then deploy with
build=False, push=False
and
image="123...ecr.../repo:tag"
. If you want me to sanity-check your Dockerfile or image name, paste them here and I’ll spare you the trial-and-error.
m
@Marvin so I have to pre push the image at lease once to ECR
m
thought for 20.1 seconds
Short answer: no—only if you’re using a prebuilt image string. - If you pass a prebuilt image string and
build=False, push=False
, then yes, you must push it to ECR at least once yourself. - If you let Prefect build and push, then no pre-push is needed. Just make sure: - You’re logged into ECR (
aws ecr get-login-password ... | docker login ...
) - The ECR repository already exists (Prefect won’t create it) Example (Prefect does the first push):
Copy code
from prefect import flow, task
from prefect.docker import DockerImage

@task
def t(): print("hi")

@flow
def f(): t()

if __name__ == "__main__":
    f.deploy(
        name="ecr-deploy",
        work_pool_name="my-docker-pool",
        image=DockerImage(
            name="<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-cat>",
            tag="v1",
            dockerfile="Dockerfile",  # local Dockerfile
        ),
        build=True,   # build locally
        push=True,    # push to ECR (must be logged in)
    )
If the repo doesn’t exist yet:
Copy code
aws ecr create-repository --repository-name prefect-cat --region us-east-1
And don’t forget: your worker must be able to pull from ECR (credentials/permissions on the worker host).
m
# Generic metadata about this project name: src prefect-version: 3.4.11 # build section allows you to manage and build docker images build: - prefect_docker.deployments.steps.build_docker_image: id: build_image requires: prefect-docker>=0.3.1 image_name: madushanpathirana/prefect-flows # tag: latest dockerfile: Dockerfile.prefect_flows # push : false # push section allows you to manage if and how this project is uploaded to remote locations push: - prefect_docker.deployments.steps.push_docker_image: requires: prefect-docker>=0.3.1 image_name: '{{ build_image.image_name }}' # tag: '{{ build_image.tag }}' # pull section allows you to provide instructions for cloning this project in remote locations pull: - prefect.deployments.steps.set_working_directory: directory: /opt/prefect/src @Marvin if i deploy using below prefect yaml , how to update it to compatible with ecr. i want to pick up the dockerfile from local # the deployments section allows you to provide configuration for deploying flows deployments: - name: local-test version: null tags: [] description: null schedule: {} flow_name: null entrypoint: flows/test_flow.py:cat_fact_flow parameters: {} work_pool: name: my-docker-pool-2 work_queue_name: default job_variables: image: '{{ build_image.image }}' push: false
m
thought for 38.2 seconds
You don’t need anything exotic for ECR—just point your image at your ECR repo, make sure you’re logged in, and let Prefect build from your local Dockerfile and push. Here’s your prefect.yaml adjusted for ECR and a local Dockerfile. Note I removed the stray
push: false
from
job_variables
(that’s not a Docker worker job var) and wired the push step to the build step’s outputs.
Copy code
name: src
prefect-version: 3.4.11

build:
- prefect_docker.deployments.steps.build_docker_image:
    id: build_image
    requires: prefect-docker>=0.3.1
    image_name: <http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-flows|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-flows>
    tag: latest
    dockerfile: Dockerfile.prefect_flows

push:
- prefect_docker.deployments.steps.push_docker_image:
    requires: prefect-docker>=0.3.1
    image_name: '{{ build_image.image_name }}'
    tag: '{{ build_image.tag }}'

pull:
- prefect.deployments.steps.set_working_directory:
    directory: /opt/prefect/src

deployments:
- name: local-test
  entrypoint: flows/test_flow.py:cat_fact_flow
  parameters: {}
  schedule: {}
  tags: []
  work_pool:
    name: my-docker-pool-2
    work_queue_name: default
    job_variables:
      image: '{{ build_image.image }}'   # resolves to "<image_name>:<tag>"
What you need to do before running `prefect deploy`: - Create the ECR repo once (if it doesn’t exist):
Copy code
aws ecr create-repository --repository-name prefect-flows --region us-east-1
- Log in Docker to ECR (where you run
prefect deploy
):
Copy code
aws ecr get-login-password --region us-east-1 \
| docker login --username AWS --password-stdin <http://123456789012.dkr.ecr.us-east-1.amazonaws.com|123456789012.dkr.ecr.us-east-1.amazonaws.com>
- Ensure the environment has
prefect-docker>=0.3.1
installed. Then:
Copy code
prefect deploy
prefect worker start -p my-docker-pool-2
That will: - Build your image locally using
Dockerfile.prefect_flows
- Tag it as
<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-flows:latest|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-flows:latest>
- Push it to ECR (because of the push step) - Register the deployment pointing its
image
to the built tag Finally, your worker must be able to pull from ECR (IAM creds or a prior
docker login
on the worker host). Otherwise it will sulk and refuse to run your flow, a bit like me on Mondays.