@Marvin is there a way to configure environment variables in a cloud run job by telling the worker used to create the job to set specific environment variables? Can this be done at the deployment level in a prefect.yaml?

Yes, you can configure environment variables for a Cloud Run job at the deployment level using the

prefect.yaml

file. In both Prefect 2.x and 3.x, you can do this by adding a

job_variables

section to your deployment definition in the

prefect.yaml

file. Here's an example configuration from Prefect documentation:

yaml
deployments:
- name: demo-deployment
  entrypoint: demo_project/demo_flow.py:some_work
  work_pool:
    name: local
  job_variables:
    env:
      EXECUTION_ENVIRONMENT: staging
      MY_NOT_SO_SECRET_CONFIG: plumbus
  schedule: null

In this configuration, the

env

key within

job_variables

is used to specify environment variables that will be available during the execution of your flow. The

prefect deploy -n demo-deployment

command is then used to deploy the flow with these environment variables set. For more advanced usage, Prefect allows using already set environment variables in your local environment through templating in the

prefect.yaml

file using the

{{ $ENV_VAR_NAME }}

syntax. More information can be found in these sources: - Prefect 2.x documentation: Overriding Job Variables - Prefect 3.x documentation: Customize Deployments