@Marvin assuming I am using prefect-docker to build and push a docker image during deployment, what is the difference between the following: 1. Using

push: True

within

prefect_docker.deployments.steps.build_docker_image

and not providing anything in the

push

section. 2. Using

push: False

within the build step and using

prefect_docker.deployments.steps.push_docker_image

in the

push

section.

Based on the information retrieved from the Prefect documentation, here are the differences between the two scenarios you've described: 1. Using

push: True

within

prefect_docker.deployments.steps.build_docker_image

and not providing anything in the

push

section means the Docker image will be automatically built and pushed to the repository referenced by the image name as part of the build step itself. 2. Using

push: False

within the build step and using

prefect_docker.deployments.steps.push_docker_image

in the

push

section means you are intentionally separating the build and push actions. This could be useful if you want to perform additional steps or checks between building the image and pushing it to the repository. Unfortunately, it seems that the extracts from the documentation do not clearly differentiate the exact behaviors of the

push

parameter within the build step and the separate

push

step scenario. Here is the relevant excerpt from the document linked below:

build: - prefect_docker.deployments.steps.build_docker_image: requires: prefect-docker>=0.3.0 image_name: my-repo/my-image tag: my-tag dockerfile: auto push: true

Once you've confirmed that these fields are set to their desired values, this step will automatically build a Docker image with the provided name and tag and push it to the repository referenced by the image name.

For more detailed and explicit information on the behaviors of both scenarios, you should refer to the official documentation: Prefect Deploy If this high-level information does not fully answer your question, I would suggest looking into the full context of the documentation or possibly searching GitHub issues for any pertinent discussions about the use of these parameters in the prefect-docker package.

the only difference is that using

push: true

is deprecated in favor of the separate pull step 🙂

@Marvin I am attempting to access a secret via the following:

secret_block: GcpSecret = GcpSecret.load("secret")
secret = secret_block.read_secret().decode("UTF-8")

This executes correctly however I am getting the pylance error

Cannot access member "decode" for type "Coroutine[Any, Any, bytes]"

on the second line since read_secret is a coroutine. What should be changed here?

In Prefect, when working with coroutines, you need to

await

the result of the coroutine to retrieve the actual value it returns. In your case,

read_secret

is a coroutine that returns

bytes

, which means you should be awaiting the completion of this coroutine to access its value before attempting to decode it. Here is the corrected code using Python's

await

to handle the coroutine:

python
secret_block: GcpSecret = GcpSecret.load("secret")

# Use `await` to wait for read_secret coroutine to return the value
secret = await secret_block.read_secret()

# Now you can decode the bytes object
decoded_secret = secret.decode("UTF-8")

Please note that you should run this within an asynchronous function because

await

can only be used within an

async def

block. Below is an excerpt from the documentation related to GCP secrets management in Prefect:

python
import os
from prefect_gcp import GCPCredentials

my_gcp_creds = GCPCredentials(
    service_account_info=os.environ.get("GCP_SERVICE_ACCOUNT_KEY_FILE_CONTENTS"),
)
my_gcp_creds.save(name="my-gcp-creds-block", overwrite=True)

Guide on Moving Data - Prefect Docs And to learn more about working with asynchronous code in Prefect, you can refer to an example from their GitHub repository. This will give you insight into handling asynchronous functions in Prefect workflows: Async Basic Python Example - GitHub Remember, if you're new to async/await in Python, these features allow for waiting for I/O bound and high-level structured network code.

@Marvin after awaiting, I am now getting the error

Argument of type "Awaitable[None]" cannot be assigned to parameter "main" of type "Coroutine[Any, Any, _T@run]" in function "run"

when calling asyncio.run on the flow from pylance

It seems that I could not find specific information related to the

asyncio.run

error you've encountered directly within Prefect's documentation or source code on GitHub. The issue might be related to Python's

asyncio

and its expected types for coroutines. To understand the problem you are facing, it usually involves attempting to run a coroutine with

asyncio.run

, and you might be passing something that isn't a coroutine. The error message you're seeing suggests that you're trying to pass an awaitable object, which is not the expected

Coroutine[Any, Any, _T@run]

type. One possible reason for this error could be that you're trying to directly call

asyncio.run

on a Prefect flow's

run

method which is not designed to be called that way. Instead, you should ensure that the flow's

run

method is awaited within an async function, and then that async function is the one passed to

asyncio.run

. Here's a sample correction:

python
import asyncio
from prefect import Flow

async def main():
    with Flow("My Flow") as flow:
        # Define your tasks and flow structure here
        pass

    # This will execute the flow run
    await flow.run()

# This is how you should pass the coroutine to asyncio.run
asyncio.run(main())

Please also note that if you are using the Prefect 2.x (Orion) version, the way of running flows might have changed. If you are indeed using Prefect 2.x, I recommend looking further into Prefect Orion's specifics, and possibly use the

prefect.engine.run

method instead, depending on the changes introduced in the newer version. You might also want to refer to the Prefect community or documentation for more details on running flows in Prefect 2.0. Always ensure to reference the latest and most relevant documentation for the version of Prefect you are using, as these things change quite frequently. Here is what I found, though it may not be precisely what you need: - engine.py from Prefect's GitHub - test_tasks.py from Prefect's GitHub Please search in the context of Prefect 2.x or Orion if that is the version causing this issue.

so if you call

SomeBlockSubclass.load()

in an async context, you need to await that call

got it, thanks @Nate

ah its pylance, yeah thats because of the

sync_compatable

decorator

Understood, thanks!

It seems there might be an issue with how the

prefect_docker.projects.steps.build_docker_image

step is processing the

requirements.txt

in your build environment. Here's some information from the documentation: The

prefect_docker.deployments.steps.build_docker_image

step should automatically build a Docker image using the provided name and tag and should install any dependencies listed in a

requirements.txt

file in the current directory. If

prefect_gcp

is specified as a dependency in your

requirements.txt

but not installed in the Docker image, consider the following potential issues: 1. The

requirements.txt

file is not in the context directory being copied into the Docker image. 2. There are permissions or path issues that prevent the

requirements.txt

from being correctly processed. 3. There is a misconfiguration in the Dockerfile or the build step that overlooks or fails to install the

requirements.txt

content. Here is the excerpt mentioning the usage of `requirements.txt`:

Prefect generates a Dockerfile for you that will build an image based off of one of Prefect's published images. The generated Dockerfile will copy the current directory into the Docker image and install any dependencies listed in a

requirements.txt

file.

Guides: Prefect Deploy To troubleshoot further: 1. Review your

requirements.txt

to ensure

prefect_gcp

is properly listed. 2. Ensure the file is in the root of the directory that is copied into the Docker image. That is often the same directory where your flow script is located. 3. Check to see if there are any steps in the Docker build process that might be failing to install the dependencies. 4. Look at the generated Dockerfile to ensure it is correctly set up to install from

requirements.txt

. 5. Try building your Docker image manually using the generated Dockerfile to see if

prefect_gcp

gets installed. If you need more specific assistance, reviewing the generated Dockerfile and the full build output could provide more clues as to why the dependencies aren't being installed as expected. If the problem persists, you may also consider checking GitHub issues for similar problems reported by other users.

@Marvin The issue seems to be occurring when the deploy script attempts to register the flow. This seems to be before the build steps are run and therefore before the requirements are added to the docker image.

The issue you are experiencing seems to stem from the flow registration step that happens before the Docker build step during the deployment process. When you run

prefect --no-prompt deploy --all

, Prefect attempts to register the flow with the Prefect backend. During registration, Prefect needs access to all the modules that are imported within your flow file. If

prefect_gcp

is not installed in the environment where

prefect deploy

is run, the registration will fail because the script cannot be executed to extract the flow due to the

ModuleNotFoundError

. To resolve this issue, you should ensure that

prefect_gcp

and any other dependencies are installed in the same environment where you're running

prefect deploy

. You can do this by: 1. Activating the environment you're using for Prefect deployment. 2. Manually installing dependencies (including

prefect_gcp

) by running

pip install -r requirements.txt

. 3. Running

prefect --no-prompt deploy --all

after all dependencies are installed. This will allow Prefect to import your flow and its dependencies without errors during the registration step, which occurs prior to building the Docker image with those same dependencies. The documentation excerpts did not provide additional detail on this specific error during deployment, but based on typical Prefect behavior, flow registration requires the code to be executable in the current environment so that the structure of the flow—its tasks, dependencies, and parameters—can be extracted and properly registered with the API. Maintaining a consistent environment for both flow registration and flow execution will prevent such module not found errors. If the resolution mentioned above does not solve your issue, you may need to look into creating a custom build step that ensures all necessary modules are installed before registration, or use a virtual environment that mirrors your target execution environment.

@Marvin I would like to use dbt core with prefect. To do so, I will need to ensure my warehouse credentials are available within the prefect execution environment. I am currently using automatic Dockerfile generation during the build_docker_image step of the deployment process. Will I need to switch that to manual generation so that I can pass the credentials file into the docker image? If so, what does the automatically generated Dockerfile contain?

The integration of dbt Core with Prefect and the management of warehouse credentials in your Prefect execution environment can be handled without necessarily switching to manual Dockerfile generation. Prefect has a feature known as "Secrets" which allows you to securely store and access sensitive information like database credentials. If you are using Prefect Cloud or Prefect Server, you can use the Prefect Secrets functionality to store and retrieve your credentials in flows. If you define these warehouse credentials as Prefect Secrets, they will be available to your flows at runtime. This method avoids having to include sensitive information directly in your Docker images or source code. Regarding the automatic Dockerfile generation provided by the

prefect_docker.projects.steps.build_docker_image

step, it typically includes: 1. A base image (usually one of Prefect's official images) 2. Instructions to copy the current project directory into the Docker image 3. Instructions to install any dependencies listed in the

requirements.txt

file located in your project directory 4. Environment variables necessary for the flow to run within Prefect's ecosystem Here's an example excerpt from the documentation regarding the automatically generated Dockerfile:

Prefect generates a Dockerfile for you that will build an image based off of one of Prefect's published images. The generated Dockerfile will copy the current directory into the Docker image and install any dependencies listed in a

requirements.txt

file.

To access your dbt Core and warehouse credentials safely within Prefect: 1. Store your warehouse credentials as secrets in Prefect Cloud or Prefect Server. 2. In your flow, retrieve these secrets using Prefect tasks designed for secret access. 3. Pass the credentials to dbt Core as required. This way, you won't need to customize the Dockerfile manually just for credentials, but you still have the option to do so if you want to include other customizations. For specific guidance on integrating dbt with Prefect, you can refer to Prefect's blogs or discussions in the community forums, as the documentation excerpts provided don't directly address the integration steps. Using Prefect with dbt Core Prefect Guides: Docker For more specific instructions on handling secrets or to understand the contents of the automatically generated Dockerfile in greater detail, you would need to consult the relevant sections of Prefect's documentation.

@Marvin the DBT blocks are not visible in my locally running UI. How to I make them appear?

To make DBT blocks appear in your locally running Prefect UI, you typically have to register the blocks with your Prefect instance. Blocks are used to store configuration or sensitive information, and in the case of DBT, you would likely store information related to project and profile configurations. If DBT blocks are not visible, it may be because: 1. They have not been registered with your Prefect instance. 2. There is a configuration issue preventing them from being displayed. To register and make the DBT blocks visible: 1. Ensure that you have created the block configuration appropriately. This usually involves defining the block's schema and filling in the necessary information, such as connection strings or file paths. The configuration syntax will depend on the type of block you're using. 2. Use the Prefect CLI or the UI to register the block with Prefect. Here is an example block registration using Prefect CLI:

shell
prefect block register -p path/to/block.yml

3. Once registered, check your Prefect UI to see if the block appears. If it doesn't, you may need to check the logs for any errors or investigate further to ensure that the block registry was successful. Unfortunately, the documentation excerpts provided do not contain specific instructions for making the DBT blocks visible in the Prefect UI. To resolve this issue, you may want to refer directly to the Prefect documentation sections on blocks, consult the Prefect community on Slack or forums for assistance, or search for issues related to block visibility on the Prefect GitHub repository. For more detailed help with Prefect blocks, visit: Prefect Docs: Blocks As this information is relatively high-level and may not fully cover your issue, I suggest searching for more specific guidance on registering and using DBT blocks within Prefect's documentation or community resources.

you can do -m prefect-dbt

Thanks @Nate. Noticed I didn't have prefect-dbt installed locally accidentally.

In DBT, a profile generally corresponds to a development, staging, or production environment. If you are utilizing DBT across three separate BigQuery projects, it is typical to create a separate DBT profile for each project. This separation ensures that each DBT profile can have its own target configuration, corresponding to the different BigQuery projects, and it can also help manage different permission levels, datasets, or other project-specific configurations. From the Prefect integrations with DBT documentation excerpts provided, you can see that the DbtCliProfile.load function is being called within a flow when utilizing Prefect with DBT:

python
@flow
def trigger_dbt_flow():
    dbt_cli_profile = DbtCliProfile.load("DBT-CORE-OPERATION-BLOCK-NAME-PLACEHOLDER")
    with DbtCoreOperation(
        commands=["dbt debug", "dbt run"],
        project_dir="PROJECT-DIRECTORY-PLACEHOLDER",
        profiles_dir="PROFILES-DIRECTORY-PLACEHOLDER",
        dbt_cli_profile=dbt_cli_profile,
    ) as dbt:
        ...

The documentation suggests you can specify the profiles directory and load the necessary profile when running DBT commands in a flow. This implies that you could use a similar pattern to specify different profiles for each BigQuery project at runtime. To accommodate multiple BigQuery projects with separate configurations using Prefect and DBT, you would: 1. Create separate DBT profiles, each configured to target a different BigQuery project. 2. Register each profile as a separate Block in Prefect. 3. Load the appropriate DBT BigQuery Target Config Block when running the flow, depending on the project you want to target. By following this approach, you can invoke DBT commands within your Prefect flows with the assurance that each BigQuery project is accessed using its respective profile configuration. For more detailed assistance, consider checking out the full Prefect + DBT integration documentation that includes code examples and explains how to set up DBT within Prefect to properly manage your projects: Prefect + DBT Integration Keep in mind to replace placeholder texts with your actual Block names and directory paths when adapting the example from the documentation.

@Marvin how do I define variables within the deployment section of the

prefect.yaml

file?

To define variables within the

prefect.yaml

file, particularly in the deployment section, you can use Prefect's templating capabilities. There are several ways to reference variables: 1. Prefect Variables: You can reference Prefect variables using the syntax

{{ prefect.variables.variable_name }}

. Prefect variables are used to store non-sensitive, reusable information, like a default image name or default work pool name. 2. Environment Variables: You can also reference environment variables using the syntax

{{ $MY_ENV_VAR }}

. This is helpful for values that are set at runtime or that need to remain flexible based on the environment where the deployment occurs. Here's an example snippet showing how to use variables in the

prefect.yaml

file:

yaml
# prefect.yaml
version: 1

# Define a preset for reuse
presets:
  defaults:
    variables:
      default_image_name: my-default-image
      work_pool: standard-pool

# Deployment section using the variables
deployments:
- name: my-deployment
  version: 1
  flow_location: flows/my_flow.py
  infrastructure:
    type: docker-container
    image: "{{ prefect.variables.default_image_name }}"
    work_pool: "{{ prefect.variables.work_pool }}"
  environment:
    # Example of setting an environment variable from the host
    MY_API_KEY: "{{ $HOST_API_KEY }}"

In this example: -

default_image_name

and

work_pool

are Prefect variables. They're defined under the

presets

section and referenced within the

infrastructure

section of the deployment. -

MY_API_KEY

is an environment variable referenced in the

environment

section of the deployment. Remember that any variables you reference in the

prefect.yaml

file need to be previously defined either as a Prefect variable or as an environment variable in the execution environment where the deployment command is run. For more information and detailed examples of how to define and use variables in

prefect.yaml

, you can refer to the following documentation: - Prefect Deploy Guide Make sure to replace

variable_name

,

MY_ENV_VAR

,

default_image_name

,

work_pool

,

MY_API_KEY

, and

HOST_API_KEY

with the actual names of the variables you want to use in your deployment.

@Marvin I have attempted to run three separate deployments using the same work pool. This has given me the error: Flow run could not be submitted to infrastructure: https://run.googleapis.com/v2/projects/project-id/locations/us-central1/jobs?jobId=prefect-infrastructure&alt=json returned "Resource 'prefect-infrastructure' already exists.". Details: "Resource 'prefect-infrastructure' already exists."> Do I need to create a new work pool for every single deployment?

No, you should not need to create a new work pool for every single deployment. Work pools in Prefect are designed to be re-usable and can handle multiple deployments. The error message you are encountering suggests that there is a conflict with a resource named "prefect-infrastructure", which already exists. As per the GitHub issue excerpt provided, it seems that there might be a situation related to provisioning infrastructure which might be causing this error. It might be the case that each deployment is attempting to create a resource with the same identifier, resulting in the "Resource 'prefect-infrastructure' already exists" error. This could be related to a naming collision within your cloud infrastructure provider (Google Cloud in this case) rather than an issue with Prefect's work pool concept itself. To solve this problem, consider the following steps: 1. Review your deployment configuration to identify if there is an infrastructure resource being defined with a fixed name "prefect-infrastructure". If so, ensure that each deployment has a unique identifier for any such resources. 2. Ensure that your deployment configurations are not attempting to create a new instance of a resource which should be re-used, like a container, virtual machine, or other infrastructure component. 3. If you have infrastructure setup scripts or automated processes as part of your deployment, check them for hardcoded resource names. You may want to look at the generated infrastructure code or scripts to check for duplicate resource names. Adjusting these to be dynamic or ensuring uniqueness can prevent this error. Documentation related to work pools suggests that work pools serve as a bridge for dynamically provisioned infrastructure, which implies they are intended to be used across multiple deployments: Work Pools - Prefect Docs For further information or to resolve issues specific to Google Cloud Run or other Prefect work pool integrations, you would benefit from checking Prefect's forums, Slack channels, or GitHub issues for similar cases or additional help from the community. Unfortunately, it is challenging to provide an exact solution without the full context of your deployment setup. If the guidance above doesn't resolve the issue, further investigation into the specifics of your Prefect and cloud provider configuration will be necessary.

@Marvin within my

prefect.yaml

I have this:

definitions:
  work_pools:
    schedules:
      daily_midnight: &daily_midnight
        cron: 0 0 * * *
    cloud_run_push_v2: &cloud_run_push_v2
      name: cloud-run-push-v2
      work_queue_name: default

deployments:
  - name: deployment-a
    entrypoint: ...
    work_pool: *cloud_run_push_v2
    schedule: *daily_midnight
  - name: deployment-b
    entrypoint: ...
    work_pool: *cloud_run_push_v2
    schedule: *daily_midnight

When I run deployment-a and deployment-b at the same time, I get that conflict error message. Can you see anything wrong here?

The conflict error message you are encountering usually occurs when there is an attempt to create two resources with the same name in the same namespace. In the context of a Prefect deployment using Google Cloud Run, this often means that there is a naming collision with the job or service being created as part of the deployment steps. In your

prefect.yaml

, you have YAML anchors (

&

) for re-usability of the

work_pools

configurations and you are referencing them in the

deployments

section. This appears to be correct in terms of YAML syntax and should not be the cause of the resource conflict. Given that you're using the same work pool (

*cloud_run_push_v2

) and the same schedule (

*daily_midnight

) for both

deployment-a

and

deployment-b

, the issue seems to be with the infrastructure on Google Cloud Run rather than your

prefect.yaml

setup. The error message suggests that when both deployments are triggered to run at midnight, they may both be trying to provision or operate on the same named resource on Google Cloud Run, hence the "Resource 'prefect-infrastructure' already exists" error. Here are some steps you can take to troubleshoot and potentially resolve the issue: 1. Review Google Cloud Run Configuration: Make sure the configuration for Google Cloud Run is set up to allow different jobs or services to run under the same resource names without conflicts. 2. Unique Naming: Ensure that each deployment yields a unique job name or service identifier on Google Cloud Run when executed. The configurations in Prefect may need to include some form of dynamic naming for each runtime instance to avoid conflicts. 3. Serialization of Runs: If the issue arises due to simultaneous executions, consider slightly offsetting the schedules or adding random jitter to stagger their start times. 4. Concurrency Limits: Look into concurrency or scaling settings in Google Cloud Run to ensure that multiple instances of your job can run concurrently within the same work pool without conflict. 5. Prefect Documentation/Support: Consult the Prefect documentation or reach out to Prefect support/community for advice regarding best practices for deploying multiple flows simultaneously with Google Cloud Run and the specific naming conventions or configurations needed to avoid this type of conflict. If none of the above steps address the conflict error, further investigation with detailed logs and full context of the deployment steps would be necessary. Given the limits of the provided excerpt, these recommendations are quite general, and more specific troubleshooting steps may be discovered by inspecting the deployment logs or Prefect's internal job creation mechanics.

@Marvin I have a number of deployments listed in

prefect.yaml

that have similar configurations with minor adjustments in the name, description, parameters & tags. Is there a way to populate the deployments section that is less repetitive?

To reduce repetition in your

prefect.yaml

deployments section, you can make use of YAML's ability to define reusable anchors (

&

) and aliases (

*

). This feature allows you to store a commonly used configuration and reference it in multiple places throughout the file. The Prefect documentation suggests adding a

definitions

section to store your aliases, which you can then use in the

deployments

section to keep it DRY (Don't Repeat Yourself). You can override specific fields of the aliased map by using the

<<: *alias

syntax and adding additional fields below. Here's an example pattern you might use:

yaml
definitions:
  work_pools:
    my_docker_work_pool: &my_docker_work_pool
      name: my-docker-work-pool
      work_queue_name: default
      # Other common job variables you want to define

deployments:
  - name: deployment-a
    <<: *my_docker_work_pool
    description: "Description for deployment-a"
    schedule: *daily_midnight
    # Any other unique parameters for deployment-a

  - name: deployment-b
    <<: *my_docker_work_pool
    description: "Description for deployment-b"
    schedule: *daily_midnight
    # Any other unique parameters for deployment-b

In this example,

definitions

is used to create an anchor for the common work pool configuration (

my_docker_work_pool

). In the

deployments

, this common configuration is referenced with aliases, and any unique configuration aspects are specified after the

<<: *alias

line. For further details and examples, refer to the Prefect documentation: - Deployment Configuration Recipes - YAML Anchors and Aliases Using this approach, you can keep the common parts of your deployment configurations centralized and avoid duplicating configuration code. Always remember to update the anchor name and any specific keys to match the context of your own

prefect.yaml

file.

@Marvin within the

definitions

section can we also configure a base deployment structure and then add overrides in the

deployments

section?

Yes, within the

definitions

section of the

prefect.yaml

file, you can configure a base deployment structure and then apply overrides in the

deployments

section. This is done by using YAML's ability to define anchors (

&

) and aliases (

*

) along with the merge key (

<<

) to extend or override the properties of the base structure. Here is an example of how you can set up a base deployment configuration in the

definitions

section and then override specific values in the

deployments

section:

yaml
definitions:
  base_deployment: &base_deployment
    work_pool: my_docker_work_pool
    schedule:
      cron: "0 0 * * *"
    tags: ["base-tag"]
    infrastructure:
      docker:
        image: "my-base-image"
    parameters:
      param1: "default-value"

deployments:
  - name: deployment-a
    <<: *base_deployment
    description: "Specific description for deployment-a"
    tags: ["deployment-a-tag"]
    # Overrides base parameter value
    parameters:
      param1: "specific-value-for-a"

  - name: deployment-b
    <<: *base_deployment
    description: "Specific description for deployment-b"
    tags: 
      - "base-tag"
      - "deployment-b-tag"  # Adds an additional tag instead of overriding the entire tags list
    # Extends base infrastructure settings
    infrastructure:
      <<: *base_deployment.infrastructure
      docker:
        image: "my-special-image-for-b"

In this example,

*base_deployment

is the alias of the reusable deployment definition, which contains common settings that can be shared across multiple deployments. When you reference this base deployment in the

deployments

section with

<<: *base_deployment

, all properties from

base_deployment

are imported. Then, for each specific deployment, you can override or extend the properties as needed: - You can add new key-value pairs or replace existing ones to change the behavior for that specific deployment. - When extending lists (like

tags

), you can list them separately to add new items without completely overriding them. - For nested structures like

infrastructure

, you can use mergers to bring in the base configuration and then specify overrides. Remember to adjust the example to fit into your actual deployment configurations, and that YAML anchors only work within the same file, so all references need to be defined within your single

prefect.yaml

. Documentation: Deployment Configuration Recipes and YAML Anchors and Aliases offer further details and examples on this usage.

@Marvin is there a way to extend the base deployment tags without having to repeat them?

Yes, there is a way to extend the base deployment tags without having to repeat them by using YAML's merge key feature (

<<

). To achieve this, you need to ensure that the tags are structured as a map of values instead of a list. Here's an example of how you can do this:

yaml
definitions:
  base_tags: &base_tags
    tag1: null
    tag2: null

deployments:
  - name: deployment-a
    tags:
      <<: *base_tags
      deployment-a-specific-tag: null

  - name: deployment-b
    tags:
      <<: *base_tags
      deployment-b-specific-tag: null

In this structure,

base_tags

is defined as a map with keys set to

null

, and we're using

&base_tags

to create an anchor for these tags. In each individual deployment, you use

<<: *base_tags

to merge the base tags with the deployment-specific tags. The keys for the tags in the individual deployments are the tag values themselves, with their own values set to

null

. While this approach mimics a list by using keys as the list contents, it might not be as intuitive and could cause confusion, since tags are typically expected to be in a list format. YAML does not directly support merging lists, so this workaround allows you to see all tags as keys in a merged map. However, this approach deviates from the usual YAML list representation for tags. Please note that YAML merging with lists is not widely supported and can also be specific to the YAML processor used by the application (such as Prefect). The syntax demonstrated here is a general representation and may not be supported directly by Prefect's deployment configuration processor. For Prefect-specific configuration, check the related documentation or consult with the Prefect community for the recommended way to handle such use cases. At the time of writing, there is not a widely supported and native way to merge lists in YAML without repeating items from the base list if custom tags (

<<: *base_tags

) are not supported in your processing environment.