In Prefect 1 0 I want to use the env parameter of the DbtShe Prefect Community #best-practices

In Prefect 1.0, I want to use the env parameter of...

Daniel Lomartra

08/29/2022, 11:16 PM

In Prefect 1.0, I want to use the env parameter of the DbtShellTask task to set environment variables that will be utilized by my dbt project. Does anyone know how this behaves if you pass env a dictionary with values that can be lists or other dictionaries? For example, if I pass it the following dictionary:

Copy code

{
	'dbt_env_var1': 'value1',
	'dbt_env_var2': ['value2','value3','value4'],
	'dbt_env_var3': {
		'another_key': 'value5'
	}
}

Will prefect accept this? will dbt_env_var2 and dbt_env_var3 be coercible back to their original data type in jinja or just passed as a string?

Daniel Lomartra

09/01/2022, 6:31 PM

I set up environment to test this. For anyone wondering about this: Prefect will error if you try to pass lists or dictionaries as values for the environment variable keys (so dbt_env_var2 and dbt_env_var3 from my example). You can, however, pass these values as a YAML string and use the fromyaml context method in jinja to parse these into the object you want (example below). You can pass the following in the env parameter within your flow:

Copy code

{
  "dbt_env_var1": "value1",
  "dbt_env_var2": "[value2,value3,value4]",
  "dbt_env_var3": "{another_key: value5}"
}

Then in dbt convert them to a the appropriate python object primitives:

Copy code

{# list #}
{{ fromyaml(env_var('dbt_env_var2')) }}
{{ fromyaml(env_var('dbt_env_var2'))[0] }}

{# dictionary #}
{{ fromyaml(env_var('dbt_env_var3')) }}
{{ fromyaml(env_var('dbt_env_var3'))['another_key'] }}

These will compile to:

Copy code

['value2', 'value3', 'value4']
value2


{'another_key': 'value5'}
value5

🙌 1

5 Views

Open in Slack

Previous Next