Daniel Lomartra

    Daniel Lomartra

    3 weeks ago
    In Prefect 1.0, I want to use the env parameter of the DbtShellTask task to set environment variables that will be utilized by my dbt project. Does anyone know how this behaves if you pass env a dictionary with values that can be lists or other dictionaries? For example, if I pass it the following dictionary:
    {
    	'dbt_env_var1': 'value1',
    	'dbt_env_var2': ['value2','value3','value4'],
    	'dbt_env_var3': {
    		'another_key': 'value5'
    	}
    }
    Will prefect accept this? will dbt_env_var2 and dbt_env_var3 be coercible back to their original data type in jinja or just passed as a string?
    I set up environment to test this. For anyone wondering about this: Prefect will error if you try to pass lists or dictionaries as values for the environment variable keys (so dbt_env_var2 and dbt_env_var3 from my example). You can, however, pass these values as a YAML string and use the fromyaml context method in jinja to parse these into the object you want (example below). You can pass the following in the env parameter within your flow:
    {
      "dbt_env_var1": "value1",
      "dbt_env_var2": "[value2,value3,value4]",
      "dbt_env_var3": "{another_key: value5}"
    }
    Then in dbt convert them to a the appropriate python object primitives:
    {# list #}
    {{ fromyaml(env_var('dbt_env_var2')) }}
    {{ fromyaml(env_var('dbt_env_var2'))[0] }}
    
    {# dictionary #}
    {{ fromyaml(env_var('dbt_env_var3')) }}
    {{ fromyaml(env_var('dbt_env_var3'))['another_key'] }}
    These will compile to:
    ['value2', 'value3', 'value4']
    value2
    
    
    {'another_key': 'value5'}
    value5