@Marvin I am looking to use the following code snippet to build a deployment. How can I define pull steps for the image?

from prefect.deployments import Deployment
from flows.train import train

train_config = {"name": "train-workflow-test",
                "path": "/opt/prefect/training/",
                "entrypoint": "flows/train.py:train",
                "work_pool_name": "work-pool-name",
                "infra_overrides": {"image": "ecr_image:0.2.0",
                                    "image_pull_policy": "Always"},
                "description": "training",
                "parameters": {"x":1, "y":2}}

deployment = await Deployment.build_from_flow(flow=train, **train_config)

To define pull steps for the image in your Prefect deployment, you can include the necessary instructions in the

prefect.yaml

file, under the pull section. The pull section contains instructions for preparing your flows for a deployment run, which are executed each time a deployment created within the specified folder is run via a worker. Here are the types of steps you could use: 1. `set_working_directory`: Sets the working directory for the process prior to importing your flow. 2. `git_clone`: Clones the repository on the specified branch. 3. `pull_from_{cloud}`: Pulls the working directory from a Cloud storage location (e.g., S3). You can also use blocks and variable references within your pull step which will remain unresolved until runtime, enabling you to fetch updated configurations each time your deployment is run without rebuilding the deployment. The following is an example of how you might use blocks to clone a repository using GitHubCredentials:

yaml
pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/org/repo.git>
      credentials: "{{ prefect.blocks.github-credentials.my-credentials }}"

Unfortunately, there's no pull step specifically for an image defined in the code snippet you've provided. If what you mean by pull steps is the

image_pull_policy

, you've already set it to "Always" in your

infra_overrides

, which instructs Kubernetes to always pull the latest image before running a job. If you're using Kubernetes as your infrastructure, then the pull policy is all you need to ensure that your specified image is pulled. For more detailed requirements, it's best to refer to the Prefect documentation or their GitHub discussions for specific use cases and configuration options. You can find the relevant documentation here.

@Marvin I am looking to use prefect sdk to build the deployment without using yaml file

Understood. If you're inclined to use Prefect's SDK to build the deployment without involving a YAML file, you indeed wouldn't need pull steps in the traditional sense that are defined within a YAML file for Prefect. The code snippet you provided is a start. If you're aiming to define steps to be executed every time the deployment is run, such as pulling the image from a Docker registry, you can specify the

image_pull_policy

within your

infra_overrides

argument to manage the image pulling behavior. The

image_pull_policy

of "Always" indicates that the image should be pulled prior to starting the container. Other possible values for

image_pull_policy

can be "IfNotPresent" or "Never", depending on your requirements. In the current SDK snippet, this configuration is already specified:

python
"infra_overrides": {"image": "ecr_image:0.2.0", "image_pull_policy": "Always"},

This means that whenever the deployment is triggered, it will always pull the image "ecr_image:0.2.0" from the container registry. Here's the revised version of your code with the appropriate configurations for image pulling without the need of a YAML file:

python
from prefect.deployments import Deployment
from flows.train import train

train_config = {
    "name": "train-workflow-test",
    "path": "/opt/prefect/training/",
    "entrypoint": "flows/train.py:train",
    "work_pool_name": "work-pool-name",
    "infra_overrides": {
        "image": "ecr_image:0.2.0",
        "image_pull_policy": "Always"  # Ensures the latest image is always used
    },
    "description": "training",
    "parameters": {"x": 1, "y": 2}
}

# Execute the deployment build process
deployment = await Deployment.build_from_flow(flow=train, **train_config)

Note that

Deployment.build_from_flow

is an asynchronous function and should be called within an async context. The Prefect SDK allows you to programmatically configure deployments. If you are within the Prefect 2.0+ domain (Orion), make sure to consult the latest Prefect documentation or GitHub repository for any updates or migration guides as the SDK and its interfaces can evolve over time.

@Marvin I am using the above code but not able to see Pull Steps under deployment configurations on prefect UI. Ideally i want to see something like following under deployment configurations/pull steps

[
  {
    "prefect.deployments.steps.set_working_directory": {
      "directory": "/opt/prefect/training"
    }
  }
]

To configure pull steps in your Prefect deployment configurations, you need to include instructions within the

prefect.yaml

file, specifically within the pull section. This section is critical as it contains the instructions for preparing your flows for a deployment run. These instructions will be executed each time a deployment created within this folder is run by a worker. There are three main types of steps you might include in the pull section: 1. set_working_directory: Sets the working directory for the process before importing your flow. 2. git_clone: Clones the provided repository on the provided branch. 3. pull_from_{cloud}: Pulls the working directory from a Cloud storage location, such as S3. Refer to Prefect Deployment Documentation for more information. If you're unable to see Pull Steps in the Prefect UI under deployment configurations, ensure that the

prefect.yaml

file is properly formatted and contains the correct YAML syntax. Also, check that the pull steps have been defined correctly within the file, as any syntax or structural error could prevent them from showing up in the UI. The example you've mentioned should appear in the

prefect.yaml

file like this:

yaml
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: "/opt/prefect/training"

Make sure your

prefect.yaml

file follows the correct format and the indentation is proper, as YAML is sensitive to indentation. If after checking the configuration issues persist, consider reaching out with more details or looking through Prefect community channels or GitHub issues for similar problems.