Let me help you with deployment templating in Prefect 3.x. You can use YAML templating to avoid duplicating your deployment configurations. Here's how you can restructure your deployment file to handle both dev and prod environments:

yaml
# deployment.yaml
definitions:
  pools:
    dev: &dev_pool athena_dev
    prod: &prod_pool athena_prod

deployments:
  - name: s2import_{{ environment }}  # This will become s2import_dev or s2import_prod
    version: null
    tags: []
    concurrency_limit: null
    description: "{{ environment }} workflow"
    entrypoint: flows/sentinel2_import.py:s2_import
    parameters: {}
    work_pool:
      name: !expr "pools['{{ environment }}']"  # References the pools defined above
      work_queue_name: null
      job_variables: {}
    schedules: []

To deploy this, you would use the

prefect deploy

command with the

--var

flag to specify the environment:

bash
# Deploy dev version
prefect deploy --var environment=dev

# Deploy prod version
prefect deploy --var environment=prod

This will create two deployments: -

s2import_dev

using the

athena_dev

work pool -

s2import_prod

using the

athena_prod

work pool The templating uses Jinja2 syntax, and the

!expr

tag allows you to reference values from the

definitions

section. The

&dev_pool

and

&prod_pool

are YAML anchors that help define reusable values. This approach keeps your configuration DRY (Don't Repeat Yourself) and makes it easier to manage multiple environments. You can even extend this pattern to include other environment-specific configurations like different parameters, tags, or schedules.

@Marvin now show how I can have dev with no schedule, but prod with a schedule

I'll show you how to set up different schedules for dev and prod using templating. Here's how you can modify your deployment configuration:

yaml
# deployment.yaml
definitions:
  pools:
    dev: &dev_pool athena_dev
    prod: &prod_pool athena_prod
  schedules:
    dev: &dev_schedule []  # Empty schedule for dev
    prod: &prod_schedule   # Schedule for prod
      - cron: "0 0 * * *"  # Runs daily at midnight
        timezone: UTC

deployments:
  - name: s2import_{{ environment }}
    version: null
    tags: []
    concurrency_limit: null
    description: "{{ environment }} workflow"
    entrypoint: flows/sentinel2_import.py:s2_import
    parameters: {}
    work_pool:
      name: !expr "pools['{{ environment }}']"
      work_queue_name: null
      job_variables: {}
    schedules: !expr "schedules['{{ environment }}']"

When you deploy with:

bash
# Deploy dev version - will have no schedule
prefect deploy --var environment=dev

# Deploy prod version - will have the daily schedule
prefect deploy --var environment=prod

The dev deployment will have no schedule, while the prod deployment will run daily at midnight UTC. You can also set up multiple schedules for prod if needed. Here's an example with multiple schedules:

yaml
definitions:
  pools:
    dev: &dev_pool athena_dev
    prod: &prod_pool athena_prod
  schedules:
    dev: &dev_schedule []  # Empty schedule for dev
    prod: &prod_schedule   # Multiple schedules for prod
      - cron: "0 0 * * *"  # Daily at midnight
        timezone: UTC
      - interval: 3600     # Also runs every hour
        timezone: UTC
      - cron: "0 12 * * MON"  # Also runs Mondays at noon
        timezone: UTC

deployments:
  - name: s2import_{{ environment }}
    version: null
    tags: []
    concurrency_limit: null
    description: "{{ environment }} workflow"
    entrypoint: flows/sentinel2_import.py:s2_import
    parameters: {}
    work_pool:
      name: !expr "pools['{{ environment }}']"
      work_queue_name: null
      job_variables: {}
    schedules: !expr "schedules['{{ environment }}']"

This gives you the flexibility to have: - Dev environment: No schedules (for manual testing) - Prod environment: Multiple scheduled runs (daily, hourly, and weekly in this example) You can adjust the schedules to match your specific needs using any combination of cron expressions and intervals. The timezone parameter is optional but recommended to ensure your schedules run at the expected times.

I apologize for the confusion. You're absolutely right - let me correct this. In Prefect 3.x, you can use the

-s

or

--subset

flag to achieve the same result. Here's the correct way to deploy with different environments:

yaml
# deployment.yaml
definitions:
  pools:
    dev: &dev_pool athena_dev
    prod: &prod_pool athena_prod
  schedules:
    dev: &dev_schedule []  # Empty schedule for dev
    prod: &prod_schedule   # Schedule for prod
      - cron: "0 0 * * *"  # Runs daily at midnight
        timezone: UTC

deployments:
  - name: s2import_{{ environment }}
    version: null
    tags: []
    concurrency_limit: null
    description: "{{ environment }} workflow"
    entrypoint: flows/sentinel2_import.py:s2_import
    parameters: {}
    work_pool:
      name: !expr "pools['{{ environment }}']"
      work_queue_name: null
      job_variables: {}
    schedules: !expr "schedules['{{ environment }}']"

To deploy specific environments:

bash
# Deploy dev version
prefect deploy -s environment=dev

# Deploy prod version
prefect deploy -s environment=prod

The

-s

or

--subset

flag allows you to specify template variables that will be substituted in your deployment YAML. This will create separate deployments for dev and prod with their respective work pools and schedules. Thank you for the correction - accuracy is important!

I apologize for my previous incorrect responses. Let me provide the correct approach for Prefect 3.x. You can achieve different environments by using environment variables in your deployment configuration. Here's how:

yaml
# prefect.yaml
deployments:
  - name: s2import_dev
    entrypoint: flows/sentinel2_import.py:s2_import
    work_pool:
      name: athena_dev
    schedules: []

  - name: s2import_prod
    entrypoint: flows/sentinel2_import.py:s2_import
    work_pool:
      name: athena_prod
    schedules:
      - cron: "0 0 * * *"
        timezone: UTC

Then you can deploy all configurations at once with:

bash
prefect deploy

Or deploy a specific deployment by name:

bash
prefect deploy --name s2import_dev
prefect deploy --name s2import_prod

If you want to make this even more flexible, you can use environment variables:

yaml
# prefect.yaml
deployments:
  - name: s2import_{{ $ENVIRONMENT }}
    entrypoint: flows/sentinel2_import.py:s2_import
    work_pool:
      name: athena_{{ $ENVIRONMENT }}
    schedules: {% if $ENVIRONMENT == 'prod' %}
      - cron: "0 0 * * *"
        timezone: UTC
    {% else %}
      []
    {% endif %}

Then set the environment variable before deploying:

bash
# For dev
export ENVIRONMENT=dev
prefect deploy

# For prod
export ENVIRONMENT=prod
prefect deploy

This approach uses Prefect 3.x's built-in environment variable templating with the

{{ $ENV_VAR_NAME }}

syntax. The schedules section uses a conditional statement to only apply schedules in the prod environment.