https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I want to adda parameter to to change...
# ask-marvin
m
@Marvin I want to adda parameter to to change the env variables in the prefect.ecs.yaml and how to change the yaml file for this prefect --no-prompt deploy --all --prefect-file prefect.ecs.yaml # Generic metadata about this project name: src prefect-version: 3.4.11 # build section allows you to manage and build docker images build: - prefect.deployments.steps.utility.run_shell_script: id: git_sha script: git rev-parse --short HEAD - prefect_docker.deployments.steps.build_docker_image: id: build_image requires: prefect-docker>=0.3.1 image_name: 010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect tag: '{{ git_sha.stdout }}' additional_tags: - dev dockerfile: Dockerfile.prefect_flows_run ignore_cache: true # push : false # push section allows you to manage if and how this project is uploaded to remote locations push: - prefect_docker.deployments.steps.push_docker_image: requires: prefect-docker>=0.3.1 image_name: '{{ build_image.image_name }}' tag: '{{ git_sha.stdout }}' additional_tags: - dev # pull section allows you to provide instructions for cloning this project in remote locations pull: - prefect.deployments.steps.set_working_directory: directory: /opt/prefect/src # the deployments section allows you to provide configuration for deploying flows deployments: - name: dev version: '{{ build_image.tag }}' tags: - ['dev'] description: null schedule: {} flow_name: null entrypoint: extraction/ebay/flow.py:run parameters: {"seller_id":"d480fd76-f59e-4ea0-95cb-4e22db0164e0","selling_partner_id":"ed442813-afe7-4026-bc14-190ca41bfb23","data_cycle":"origination"} work_pool: name: my-ecs-pool work_queue_name: default job_variables: image: 010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{ git_sha.stdout }} # launch_type: EC2 cpu: 4096 # vCPU memory: 8192 # MiB push: false env: APP_ENV: "dev" SOURCE : "ebay"
m
✅ thought for 759.9 seconds
Great question — there are a few good ways to make the 
env
values in your 
prefect.ecs.yaml
configurable in Prefect 3.x. Below are the options and examples. First, two quick notes about your YAML: - For ECS workers, 
work_pool.job_variables.env
should be a dict (KEY: VALUE), which you’re already using — good. - Your 
tags
should be a list of strings (not a nested list) and the 
image
line looks malformed. Also, 
push: false
is not a valid job variable and should be removed from 
job_variables
. Ways to parameterize env for dev/stage/prod 1) Define multiple deployments in one file (works with --all) - Best if you want 
prefect --no-prompt deploy --all --prefect-file prefect.ecs.yaml
to create all envs at once. Example:
Copy code
# Generic metadata about this project
name: src
prefect-version: 3.4.11

build:
  - prefect.deployments.steps.utility.run_shell_script:
      id: git_sha
      script: git rev-parse --short HEAD
  - prefect_docker.deployments.steps.build_docker_image:
      id: build_image
      requires: prefect-docker>=0.3.1
      image_name: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect>
      tag: '{{ git_sha.stdout }}'
      additional_tags:
        - dev
      dockerfile: Dockerfile.prefect_flows_run
      ignore_cache: true

push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.3.1
      image_name: '{{ build_image.image_name }}'
      tag: '{{ git_sha.stdout }}'
      additional_tags:
        - dev

pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /opt/prefect/src

deployments:
  - name: dev
    version: '{{ build_image.tag }}'
    tags: ['dev']
    entrypoint: extraction/ebay/flow.py:run
    parameters:
      seller_id: "d480fd76-f59e-4ea0-95cb-4e22db0164e0"
      selling_partner_id: "ed442813-afe7-4026-bc14-190ca41bfb23"
      data_cycle: "origination"
    work_pool:
      name: my-ecs-pool
      work_queue_name: default
      job_variables:
        image: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{> git_sha.stdout }}
        cpu: 4096
        memory: 8192
        env:
          APP_ENV: "dev"
          SOURCE: "ebay"

  - name: stage
    version: '{{ build_image.tag }}'
    tags: ['stage']
    entrypoint: extraction/ebay/flow.py:run
    parameters:
      seller_id: "d480fd76-f59e-4ea0-95cb-4e22db0164e0"
      selling_partner_id: "ed442813-afe7-4026-bc14-190ca41bfb23"
      data_cycle: "origination"
    work_pool:
      name: my-ecs-pool
      work_queue_name: default
      job_variables:
        image: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{> git_sha.stdout }}
        cpu: 4096
        memory: 8192
        env:
          APP_ENV: "stage"
          SOURCE: "ebay"

  - name: prod
    version: '{{ build_image.tag }}'
    tags: ['prod']
    entrypoint: extraction/ebay/flow.py:run
    parameters:
      seller_id: "d480fd76-f59e-4ea0-95cb-4e22db0164e0"
      selling_partner_id: "ed442813-afe7-4026-bc14-190ca41bfb23"
      data_cycle: "origination"
    work_pool:
      name: my-ecs-pool
      work_queue_name: default
      job_variables:
        image: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{> git_sha.stdout }}
        cpu: 4096
        memory: 8192
        env:
          APP_ENV: "prod"
          SOURCE: "ebay"
Then:
Copy code
prefect --no-prompt deploy --all --prefect-file prefect.ecs.yaml
2) Use Prefect Variables in your YAML - Define variables once and reference them in the YAML. - Good if you want to change values without touching the repo. YAML:
Copy code
env:
  APP_ENV: "{{ prefect.variables.app_env }}"
  SOURCE: "{{ prefect.variables.source }}"
Set variables:
Copy code
prefect variable set app_env dev
prefect variable set source ebay
Docs: Prefect Variables and prefect.yaml templating
3) Use shell environment variables during deploy - Interpolate from your shell env at deploy-time. YAML:
Copy code
env:
  APP_ENV: "{{ $APP_ENV }}"
  SOURCE: "{{ $SOURCE }}"
Deploy:
Copy code
export APP_ENV=stage SOURCE=ebay
prefect --no-prompt deploy --all --prefect-file prefect.ecs.yaml
Docs: prefect.yaml (environment variable templating with 
{{ $VAR }}
) 4) CLI overrides with --job-variable (individual deployments only) - Use this when deploying a single deployment (not with --all). Examples:
Copy code
prefect deploy --prefect-file prefect.ecs.yaml --name dev \
  --job-variable env.APP_ENV=dev \
  --job-variable env.SOURCE=ebay

# or short form
prefect deploy --prefect-file prefect.ecs.yaml -n dev \
  -jv env.APP_ENV=stage -jv env.SOURCE=ebay
Note: 
--job-variable
overrides do not apply when using 
--all
. You must deploy a single deployment by name. Docs: Customize job variables Common fixes to your current file - Fix tags and image line; remove the stray 
push: false
under 
job_variables
. - Quote templated expressions: - 
image: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{> git_sha.stdout }}
- 
env
values can be quoted: 
"{{ $APP_ENV }}"
, 
"{{ prefect.variables.app_env }}"
Questions for you - Do you want to keep using 
--all
to create dev/stage/prod in one go? If yes, I recommend option 1 (define three deployments in the same file). - If you prefer a single deployment and change env per deploy, do you want to pass values via shell env (option 3) or via 
--job-variable
flags (option 4)? Relevant docs - prefect.yaml templating: prefect.yaml - Customize job variables (including CLI overrides): Customize job variables - Prefect Variables: Prefect Variables
2 Views
Open in Slack
PreviousNext
Powered by