Ronald Sam
08/17/2023, 7:14 PMMarvin
08/17/2023, 7:15 PMprefect.yaml
file, you can specify the infra_overrides
option. This is used to override the default infrastructure settings for your Prefect deployments.
Here's a basic example of how you might use it:
yaml
# prefect.yaml
deployments:
- name: my-flow-deployment
infra_overrides:
- option: docker-container
value: s3/my-bucket
In this example, the infra_overrides
option is used to specify that the docker-container
should be overridden with the value s3/my-bucket
.
Remember, the infra_overrides
option is just one of many ways to customize your Prefect deployments. You can also use the prefect deploy
CLI command to override deployment configurations.
Oh, and one more thing. Always double-check your prefect.yaml
file for any syntax errors. They're like little gremlins that love to cause trouble.
If you need more detailed information, you might want to visit the Prefect documentation here and here. It's a thrilling read, I assure you.Nate
08/17/2023, 7:49 PMjob_variables
are now the way we can override a work pool on a deployment basis. you should be able to find an example in the docs somewhereMarvin
08/17/2023, 7:50 PMjob_variables
option in the prefect.yaml
file. My apologies for the confusion. It's truly a testament to the ever-evolving nature of technology, isn't it?
Here's an example of how you might use job_variables
to override a work pool on a deployment basis:
yaml
# prefect.yaml
deployments:
- name: my-flow-deployment
job_variables:
WORK_POOL: my-custom-work-pool
In this example, the WORK_POOL
variable is overridden with the value my-custom-work-pool
for the my-flow-deployment
deployment. This allows you to specify a different work pool for each deployment, which can be quite handy.
Remember, the prefect.yaml
file is like a blueprint for your Prefect deployments. Any changes you make here will affect how your deployments are configured and run. So, handle with care!
If you need more detailed information, you might want to visit the Prefect documentation here and here. It's a thrilling read, I assure you.Nate
08/17/2023, 7:50 PMdeployments:
- name: chroma-healthcheck
entrypoint: examples/flows/chroma_healthcheck.py:test_connection
work_pool:
name: kubernetes-prd-internal-tools
job_variables:
env:
MARVIN_OPENAI_API_KEY: "{{ prefect.blocks.secret.openai-api-key }}"
MARVIN_CHROMA_SERVER_HOST: "{{ $MARVIN_CHROMA_SERVER_HOST }}"
MARVIN_CHROMA_SERVER_HTTP_PORT: "{{ $MARVIN_CHROMA_SERVER_HTTP_PORT }}"
Marvin
08/17/2023, 7:51 PMjob_variables
option in the prefect.yaml
file. My previous explanation was a bit too general, wasn't it? I apologize for that. Let's dive into your example:
yaml
deployments:
- name: chroma-healthcheck
entrypoint: examples/flows/chroma_healthcheck.py:test_connection
work_pool:
name: kubernetes-prd-internal-tools
job_variables:
env:
MARVIN_OPENAI_API_KEY: "{{ prefect.blocks.secret.openai-api-key }}"
MARVIN_CHROMA_SERVER_HOST: "{{ $MARVIN_CHROMA_SERVER_HOST }}"
MARVIN_CHROMA_SERVER_HTTP_PORT: "{{ $MARVIN_CHROMA_SERVER_HTTP_PORT }}"
In this example, the job_variables
option is used to set environment variables for the chroma-healthcheck
deployment. Specifically, it sets the MARVIN_OPENAI_API_KEY
, MARVIN_CHROMA_SERVER_HOST
, and MARVIN_CHROMA_SERVER_HTTP_PORT
environment variables.
The values for these variables are fetched from Prefect secrets and environment variables, as indicated by the {{ prefect.blocks.secret.openai-api-key }}
and {{ $MARVIN_CHROMA_SERVER_HOST }}
syntax.
This allows you to dynamically configure your deployments based on the environment in which they're running. It's a powerful feature, but with great power comes great responsibility. So, handle with care!
I hope this clarifies things for you. If not, feel free to ask more questions. I'm here to help, after all. Well, at least that's what they tell me.Ronald Sam
08/17/2023, 7:52 PM- name: test_2_stg
version:
tags: []
description:
schedule: {}
flow_name:
entrypoint: .\AppScripts\PrefectDemo\demo.py:my_favorite_function
parameters: {}
work_pool: *aws_stg_work_pool
job_variables:
infra_overrides: { command: "" }
pull:
- prefect.deployments.steps.set_working_directory:
directory: Z:\git\DevOps
- prefect.deployments.steps.pip_install_requirements:
directory: Z:\git\DevOps\AppScripts\PrefectDemo
requirements_file: requirements.txt
stream_output: true
Nate
08/17/2023, 7:53 PMcommand
is a job_variable
that you might override on a work pool for a given deployment- name: test_2_stg
version:
tags: []
description:
schedule: {}
flow_name:
entrypoint: .\AppScripts\PrefectDemo\demo.py:my_favorite_function
parameters: {}
work_pool: *aws_stg_work_pool
job_variables:
command: "echo hello"
Ronald Sam
08/17/2023, 7:53 PMNate
08/17/2023, 7:54 PMdeployments:
- name: foo
entrypoint: src/demo_project/healthcheck.py:healthcheck
schedule: {}
parameters:
message: "Don't panic!"
work_pool:
<<: *local_work_pool
job_variables:
env:
FOO: bar
where
definitions:
work_pools:
k8s_work_pool: &k8s_work_pool
name: k8s
work_queue_name: default
job_variables:
image: "{{ build_image.image }}"
local_work_pool: &local_work_pool
name: local
Ronald Sam
08/17/2023, 8:04 PM- name: test_2_stg
version:
tags: []
description:
schedule: {}
flow_name:
entrypoint: .\AppScripts\PrefectDemo\demo.py:my_favorite_function
parameters: {}
work_pool: *aws_stg_work_pool
infra_overrides:
{"command": "Z:\\git\\DevOps\\AppScripts\\PrefectDemo\\.venv\\Scripts\\python -m prefect.engine"}
pull:
- prefect.deployments.steps.set_working_directory:
directory: Z:\git\DevOps
- prefect.deployments.steps.pip_install_requirements:
directory: Z:\git\DevOps\AppScripts\PrefectDem\o
requirements_file: requirements.txt
stream_output: true
Nate
08/17/2023, 8:12 PMinfra_overrides
key here anymore (anymore meaning since the introduction of prefect.yaml and prefect deploy
)
the things you used to override with infra_overrides
like cpu, memory, command, etc are all just fields on a work pool, or to say it differently, job_variables
you will not provide a infra_overrides
key in your prefect.yaml
, you can accomplish the same by providing job_variables
to override the default fields on a work pool like this, this or this - does that make sense?Ronald Sam
08/17/2023, 8:13 PMNate
08/17/2023, 8:13 PM- name: foo
entrypoint: src/demo_project/healthcheck.py:healthcheck
schedule: {}
parameters:
message: "Don't panic!"
work_pool:
<<: *local_work_pool
job_variables:
env:
FOO: bar
and now I haveRonald Sam
08/17/2023, 8:13 PMNate
08/17/2023, 8:16 PMRonald Sam
08/17/2023, 8:16 PM