I want to apply settings such as <https docs prefect io 2 11 Prefect Community #ask-community

I want to apply settings such as <PREFECT_LOGGING_...

Martin T

08/08/2023, 1:51 PM

I want to apply settings such as PREFECT_LOGGING_EXTRA_LOGGERS for Prefect Cloud flows. But the docs on Profiles and Configuration seems to have been written with a dev/localhost use-case in mind. What is the recommended way to deploy these settings? Can it be specified in my

prefect.yaml

deployments?

✅ 1

Christopher Boyd

08/08/2023, 1:55 PM

They can be passed as env vars or if in a container with a logging configuration specified

Christopher Boyd

08/08/2023, 1:56 PM

https://docs.prefect.io/2.11.3/guides/logs/#logging-configuration

Christopher Boyd

08/08/2023, 1:57 PM

It would need to be updated in the container if that’s what you’re looking for though , as once the process starts it would not see any subsequent changes

Christopher Boyd

08/08/2023, 1:58 PM

Env variables are job overrides which can be set in prefect yaml

Martin T

08/08/2023, 1:59 PM

But how do I set env vars for a flow in Prefect 2.0? In 1.0 we did:

Copy code

DockerRun(
     env={
         "PREFECT__LOGGING__LEVEL": "DEBUG",
         "PREFECT__LOGGING__EXTRA_LOGGERS": "['my_package']",

Martin T

08/08/2023, 2:05 PM

"Env variables are job overrides which can be set in prefect yaml" where is this syntax specified?

Christopher Boyd

08/08/2023, 2:07 PM

https://docs.prefect.io/2.11.3/concepts/deployments/#templating-options

Christopher Boyd

08/08/2023, 2:10 PM

There is a section there for referencing env vars like ‘{{ $PREFECT_API_URL }}’ which is passed your flow run

Martin T

08/08/2023, 2:14 PM

Sorry for being a bit slow here 🙂 I know how to referece local env var at build/deploy time. But I need to SET env vars, not reference local ones. Is it something like this?

Copy code

deployments:
    ....
    parameters: {ENV VARS HERE?}

Martin T

08/08/2023, 2:15 PM

No soryy, parameters are of course what's passed to the flow function.

Christopher Boyd

08/08/2023, 2:15 PM

The job overrides pass environment variables that are assumed by the flow

Christopher Boyd

08/08/2023, 2:16 PM

If you want an environment accessible by the flows environment , that’s a job variable

Christopher Boyd

08/08/2023, 2:16 PM

Whether it’s local or remote is irrelevant , it can be one you dynamically source from a runtime config or branch like in GitHub action, or one you statically pass like a debug mode

Christopher Boyd

08/08/2023, 2:18 PM

https://docs.prefect.io/2.11.3/api-ref/prefect/deployments/deployments/

Christopher Boyd

08/08/2023, 2:19 PM

Job variables in Prefect.yaml maps to the infra overrides in the deployment class

Martin T

08/08/2023, 2:25 PM

So it's the infra_overrides
setting? Can you complte this example?

Copy code

...
deployments:
- name: dev
  entrypoint: src/flows/some_flow.py:my_flow
  version: "{{ build-image.tag }}"
  work_pool: *pool
  schedule: false
  infra_overrides: ????

Christopher Boyd

08/08/2023, 2:27 PM

job_variables: env: PREFECT__LOGGING__LEVEL: “DEBUG”_

Christopher Boyd

08/08/2023, 2:28 PM

The kwarg / parameter is job variables , not infra overrides

Christopher Boyd

08/08/2023, 2:28 PM

That’s what it maps back to for the Deployment class

Christopher Boyd

08/08/2023, 2:29 PM

https://docs.prefect.io/2.11.3/concepts/deployments/#work-pool-fields

Martin T

08/08/2023, 2:45 PM

Now my prefect.yaml looks like this, where I have tried to raise the debug level to INFO.

Copy code

# prepare steps:
build:
- prefect_docker.deployments.steps.build_docker_image:
    id: build-image
    requires: prefect-docker>=0.3.0
    image_name: <LOCAL_REPO_IMAGE>
    tag: "{{ $VERSION }}"
    dockerfile: Dockerfile.prefect

push:
- prefect_docker.deployments.steps.push_docker_image:
    requires: prefect-docker>=0.3.0
    image_name: "{{ build-image.image_name }}"
    tag: "{{ build-image.tag }}"

# runtime steps
pull:
- prefect.projects.steps.set_working_directory:
    directory: /opt/prefect

definitions:
  pool: &pool
    name: default-docker-pool
    job_variables:
      image: "{{ build-image.image }}"
      image_pull_policy: Always
      env:
        PREFECT_LOGGING__LEVEL: "INFO"

deployments:

- name: dev
  entrypoint: src/flow.py:my_flow
  version: "{{ build-image.tag }}"
  work_pool: *pool
  schedule: false
  parameters:
    foo: bar

But when deployed it still logs DEBUG as well

Martin T

08/08/2023, 2:48 PM

Christopher Boyd

08/08/2023, 2:51 PM

I’d start with something else just to make sure the variables are passing through right

Christopher Boyd

08/08/2023, 2:52 PM

Like just setting a key value and printing / retrieving it in your flow to confirm it’s there .

Christopher Boyd

08/08/2023, 2:53 PM

You also have some other settings on your worker or in your environment - there is both a debug mode flag and a logging level flag

Christopher Boyd

08/08/2023, 3:00 PM

It also looks like the syntax is off

Christopher Boyd

08/08/2023, 3:00 PM

You have a single underscore and a double underscore both in the key

Martin T

08/08/2023, 3:20 PM

Yes the double underscore was wrong (I believe it was correct syntax in 1.0 as it was copy-pasted from the docs originally, but leave that be.) Finally working, thanks for your patience.

Copy code

...
definitions:
  pool: &pool
    name: default-docker-pool
    job_variables:
      image: "{{ build-image.image }}"
      image_pull_policy: Always
      env:
        PREFECT_LOGGING_LEVEL: "DEBUG"
        PREFECT_LOGGING_EXTRA_LOGGERS: "..."

deployments:

- name: dev
  entrypoint: src/.../flow.py:my_flow
  version: "{{ build-image.tag }}"
  work_pool: *pool
  schedule: false
  parameters:
    ...: ...

✅ 1

🙌 1

3 Views

Open in Slack

Previous Next