https://prefect.io logo
Channels
Powered by
m

Martin T

08/08/2023, 1:51 PM
I want to apply settings such as PREFECT_LOGGING_EXTRA_LOGGERS for Prefect Cloud flows. But the docs on Profiles and Configuration seems to have been written with a dev/localhost use-case in mind. What is the recommended way to deploy these settings? Can it be specified in my 
prefect.yaml
deployments?
✅ 1
c

Christopher Boyd

08/08/2023, 1:55 PM
They can be passed as env vars or if in a container with a logging configuration specified
It would need to be updated in the container if that’s what you’re looking for though , as once the process starts it would not see any subsequent changes
Env variables are job overrides which can be set in prefect yaml
m

Martin T

08/08/2023, 1:59 PM
But how do I set env vars for a flow in Prefect 2.0? In 1.0 we did:
Copy code
DockerRun(
     env={
         "PREFECT__LOGGING__LEVEL": "DEBUG",
         "PREFECT__LOGGING__EXTRA_LOGGERS": "['my_package']",
"Env variables are job overrides which can be set in prefect yaml" where is this syntax specified?
c

Christopher Boyd

08/08/2023, 2:07 PM
There is a section there for referencing env vars like ‘{{ $PREFECT_API_URL }}’ which is passed your flow run
m

Martin T

08/08/2023, 2:14 PM
Sorry for being a bit slow here 🙂 I know how to referece local env var at build/deploy time. But I need to SET env vars, not reference local ones. Is it something like this?
Copy code
deployments:
    ....
    parameters: {ENV VARS HERE?}
No soryy, parameters are of course what's passed to the flow function.
c

Christopher Boyd

08/08/2023, 2:15 PM
The job overrides pass environment variables that are assumed by the flow
If you want an environment accessible by the flows environment , that’s a job variable
Whether it’s local or remote is irrelevant , it can be one you dynamically source from a runtime config or branch like in GitHub action, or one you statically pass like a debug mode
Job variables in Prefect.yaml maps to the infra overrides in the deployment class
m

Martin T

08/08/2023, 2:25 PM
So it's the 
infra_overrides
 setting? Can you complte this example?
Copy code
...
deployments:
- name: dev
  entrypoint: src/flows/some_flow.py:my_flow
  version: "{{ build-image.tag }}"
  work_pool: *pool
  schedule: false
  infra_overrides: ????
c

Christopher Boyd

08/08/2023, 2:27 PM
job_variables: env: PREFECT__LOGGING__LEVEL: “DEBUG”_
The kwarg / parameter is job variables , not infra overrides
That’s what it maps back to for the Deployment class
m

Martin T

08/08/2023, 2:45 PM
Now my prefect.yaml looks like this, where I have tried to raise the debug level to INFO.
Copy code
# prepare steps:
build:
- prefect_docker.deployments.steps.build_docker_image:
    id: build-image
    requires: prefect-docker>=0.3.0
    image_name: <LOCAL_REPO_IMAGE>
    tag: "{{ $VERSION }}"
    dockerfile: Dockerfile.prefect

push:
- prefect_docker.deployments.steps.push_docker_image:
    requires: prefect-docker>=0.3.0
    image_name: "{{ build-image.image_name }}"
    tag: "{{ build-image.tag }}"

# runtime steps
pull:
- prefect.projects.steps.set_working_directory:
    directory: /opt/prefect

definitions:
  pool: &pool
    name: default-docker-pool
    job_variables:
      image: "{{ build-image.image }}"
      image_pull_policy: Always
      env:
        PREFECT_LOGGING__LEVEL: "INFO"

deployments:

- name: dev
  entrypoint: src/flow.py:my_flow
  version: "{{ build-image.tag }}"
  work_pool: *pool
  schedule: false
  parameters:
    foo: bar
But when deployed it still logs DEBUG as well
image.png
c

Christopher Boyd

08/08/2023, 2:51 PM
I’d start with something else just to make sure the variables are passing through right
Like just setting a key value and printing / retrieving it in your flow to confirm it’s there .
You also have some other settings on your worker or in your environment - there is both a debug mode flag and a logging level flag
It also looks like the syntax is off
You have a single underscore and a double underscore both in the key
m

Martin T

08/08/2023, 3:20 PM
Yes the double underscore was wrong (I believe it was correct syntax in 1.0 as it was copy-pasted from the docs originally, but leave that be.) Finally working, thanks for your patience.
Copy code
...
definitions:
  pool: &pool
    name: default-docker-pool
    job_variables:
      image: "{{ build-image.image }}"
      image_pull_policy: Always
      env:
        PREFECT_LOGGING_LEVEL: "DEBUG"
        PREFECT_LOGGING_EXTRA_LOGGERS: "..."

deployments:

- name: dev
  entrypoint: src/.../flow.py:my_flow
  version: "{{ build-image.tag }}"
  work_pool: *pool
  schedule: false
  parameters:
    ...: ...
✅ 1
🙌 1
Powered by