@Marvin How to use single docker image for multiple deployments?

According to the Prefect documentation, you can manage multiple deployments by reusing the same Docker image with different configurations within the same codebase. To ensure proper reuse, dynamic variables such as

{{ build-image.image }}

can be populated with relevant image names and tags during the deployment process. This allows you to deploy a new version of your deployment every time with the updated image details. The

prefect.yaml

file supports multiple deployment declarations which enable the configuration for all deployments to be version-controlled and released with a single command. Each deployment declaration must have a unique

name

field to differentiate them. Here are some key points from the documentation: - Use

prefect init

with the

docker

recipe and appropriate fields to set the image name and tag. - Utilize

prefect_docker.deployments.steps.build_docker_image

to build and push the Docker image with the specified name and tag. - Leverage template values like

{{ image }}

within

prefect.yaml

to avoid hardcoding values and to keep the build step and deployment specification in sync. For more detailed information, please refer to the Prefect Deploy documentation.

@Marvin each time I run

prefect --no-prompt deploy --name my-deploy-*

multiple deployments are run. All those deployments have the same

prefect_docker.deployments.steps.build_docker_image

step. However, it is repeated for each deployment.

The Prefect documentation suggests that you can dynamically populate the image details using variables like

{{ build-image.image }}

to avoid repeating the build step for each deployment. This implies that if you set up the build step correctly to produce fields called

image_name

and

tag

, you can reference these values in your deployments, thus ensuring the build step is not unnecessarily repeated for each deployment. However, it appears the documentation does not provide explicit details on how to ensure the

prefect_docker.deployments.steps.build_docker_image

step is executed only once when deploying multiple configurations with the same Docker image. One approach to address this issue might be to build the Docker image separately and push it to your image registry before running the

prefect deploy

command. This way, instead of building the image as part of each deployment step, you can reference the already built Docker image in your

prefect.yaml

configurations. For further guidance or to suggest improvements, consider opening a discussion or issue in the Prefect GitHub repository. Additionally, if this feature is not currently supported, reaching out to Prefect through their GitHub Discussions might illuminate best practices or upcoming features that could address your concerns. For now, checking out the deployments tutorial might also offer additional insights into managing deployments efficiently.

@Marvin Can I skip build_docker_image step based on run_shell_script step?

The Prefect documentation does not explicitly indicate that you can skip the

build_docker_image

step based on the result of a

run_shell_script

step directly within the

prefect.yaml

file or during a deployment command. The build steps typically run in the order that they are provided within the

prefect.yaml

file. However, the documentation does mention that build, push, and pull sections can be overridden on a per-deployment basis by defining

build

,

push

, and

pull

fields within a deployment definition in the

prefect.yaml

file. This allows you to customize the behavior of your deployments if they require different instructions. To conditionally skip the

build_docker_image

step, you would need to implement logic outside of the

prefect.yaml

and deployment process, such as with a separate script that conditionally triggers the Prefect CLI commands based on the conditions or results you define. For scenarios where the Prefect documentation does not provide a direct solution, a common approach is to create a custom build script that performs the desired checks and conditionally executes the

prefect deploy

command with the necessary build steps. For more information on overriding deployment instructions and to see if the deployment action for the build section would support conditional logic, please refer to the Prefect Deploy Guide.

@Marvin how to override a part of build section on per-deployment basis?

To override a part of the build section on a per-deployment basis within the

prefect.yaml

file, you can define

build

,

push

, and

pull

fields specifically within a deployment's definition. You can utilize this capability to customize deployment instructions for multiple deployments that require differentiated deployment steps. Here's how you can do it: 1. Within your

prefect.yaml

file, locate the deployment declaration you want to customize. 2. Add the

build

,

push

, or

pull

fields as needed within the deployment definition to override the default instructions provided at the global level. The

prefect deploy

command will use the

build

,

push

, or

pull

instructions that are provided within the deployment's definition in the

prefect.yaml

file. For instance:

yaml
deployments:
  - name: custom-deployment-1
    # ...other deployment configurations...
    build:
      - step: custom-step-for-this-deployment
        # ...custom step details...

By including these overrides within the deployment definition, you can ensure that the specified deployment runs the unique steps required for that particular deployment's circumstances, independent of the other deployments defined in the

prefect.yaml

file. For additional information and concrete examples, refer to the Prefect Deploy Guide documentation.

@Marvin Can I override only a few parameters in prefect_docker.deployments.steps.build_docker_image step?

According to the documentation, you can override parameters for specific deployment steps, including the

prefect_docker.deployments.steps.build_docker_image

step, by defining the appropriate fields within a deployment definition in the

prefect.yaml

file. When you run the

prefect deploy

command, it will utilize any

build

,

push

, or

pull

instructions provided in the deployment's definition. This means that if you want to override only a few parameters of the

build_docker_image

step for a particular deployment, you should specify those parameters within the

build

section of that deployment's configuration in the

prefect.yaml

file. For example, if you wanted to override the

tag

and

push

parameters for a specific deployment, your

prefect.yaml

file might look something like this:

yaml
deployments:
  - name: custom-deployment
    # ...other deployment configurations...
    build:
      - prefect_docker.deployments.steps.build_docker_image:
          image_name: my-repo/my-custom-image
          tag: my-custom-tag
          push: false
          # other parameters remain as defined in the global build section

In this example, only the

tag

and

push

parameters are overridden for

custom-deployment

, while other parameters not mentioned will use their default values as specified in the global

build

section or from the step definition. For complete guidance and additional examples on overriding deployment steps, please refer to the Prefect Deploy documentation.

Hi @Anton L., I am trying to do the same thing - when running a

prefect deploy --all

I would like for it to build the docker image once. Did you convince Prefect to only build/push the container once?

hi all this is a known limitation of

prefect deploy

if you want to reuse the same image a bunch, i would recommend just building your

Dockerfile

in CI as you might otherwise and setting the resulting

image

in the

job_variables

section if you dont want to use a static image tag like latest, you could use a common reference like a git sha

Thanks Nate. That is a bummer. I like the date/time default tag that prefect uses when you omit the image tag. 🙂

I built the image in a first deployment and override build: null for all others. also, I hardcoded image name and tag in push step

Is there a way to get prefect to execute only the build step? its convenient to have it do that work (once)

today if a

build

step exists for a given deployment definition (no explicit or implicit/global build step) then it will run along with the

push

step at deployment time, for each such definition we'd like to add a

--build-once

flag someday

I am hoping to have dozens of flows deployed over time. --build-once seems pretty important. But hey, Prefect so far is very sweet I am not complaining too loudly. Maybe I can get one of my guys to submit a PR

so, i had a bit of an idea on this I can look more later (the cache key thing could be improved), but this is working using this custom step +

prefect --no-prompt deploy --all

trying to absorb the yaml here…

ok gotcha. not sure if I can/would go down this route

yeah Im showing more of the guts here than we would in anything we'd release, just in case you were curious in having someone open a PR

I think getting my CI to run the

docker build

command direct, then not having a build in the prefect.yaml is where I am going to go. Too bad since centralizing things like the image tag in one place was nice. That reminds me I am going to have to propagate whatever tag I land on from

docker build

to

prefect deploy

. I guess date/time isn’t going to work well. 🤔

you can template in env vars in the

prefect.yaml

->

image

so for example if you're on GHA you can

some_var >> GITHUB_OUTPUT

and then set that in the env of the step where you run your

prefect deploy

like

Seems reasonable.