https://prefect.io logo
Join Slack
Channels
Powered by Linen
Title
c

Constantino Schillebeeckx

08/06/2021, 5:56 PM
hi prefect! I"m struggling to use the 
DbtShellTask
along with secrets (more details in thread)
I first tried the following (see screenshot), but I got the stacktrace: 
Task 'DbtShellTask': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 865, in get_task_run_state
    logger=self.logger,
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 328, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 441, in method
    return run_method(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/dbt/dbt.py", line 183, in run
    command=command, env=env, helper_script=helper_script
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 441, in method
    return run_method(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/shell.py", line 140, in run
    shell=sys.platform == "win32",
  File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.7/subprocess.py", line 1462, in _execute_child
    env_list.append(k + b'=' + os.fsencode(v))
  File "/usr/local/lib/python3.7/os.py", line 812, in fsencode
    filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not Secret
so then I tried the following, but then got 
Failed to load and execute Flow's environment: ValueError('Secrets should only be retrieved during a Flow run, not while building a Flow.')
what's the proper way to set the 
DbtShellTask
env with secrets? cc @Kevin Kho
here's another approach that fails 
Task 'DbtShellTask': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 865, in get_task_run_state
    logger=self.logger,
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 328, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 441, in method
    return run_method(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/dbt/dbt.py", line 183, in run
    command=command, env=env, helper_script=helper_script
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 441, in method
    return run_method(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/shell.py", line 140, in run
    shell=sys.platform == "win32",
  File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.7/subprocess.py", line 1462, in _execute_child
    env_list.append(k + b'=' + os.fsencode(v))
  File "/usr/local/lib/python3.7/os.py", line 812, in fsencode
    filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not PrefectSecret
e

emre

08/06/2021, 7:21 PM
In all of your approaches, you are trying to feed the dbt task your credentials in 
__init__
time, while they are available in 
run
time. Your last example uses 
PrefectSecret
tasks to fetch credentials. That is a 
Task
and therefore it will fetch the credentials when you do 
flow.run()
, not within 
with Flow():
. However, you are passing these tasks to 
DbtShellTask.__init__
under the 
env
parameter.
i.e., try moving the 
env
parameter from the first set of parentheses (
__init__
) toı the second one, which will correctly set the credentials as an upstream dependency for the runtime. 
DbtShellTask(profiles_dir='', ...)(
    command='dbt run',
    env = {
        'USERNAME': PrefectSecret('User'),
        'PASSWORD': PrefectSecret('Pass')
    }
)
k

Kevin Kho

08/06/2021, 7:46 PM
Hey @Constantino Schillebeeckx, am out of office today so will just chime in a bit but @emre is right here. You can either use 
Secret("xxx").get()
to get it in the first example. Emre’s example seems like the easiest approach. It defers loading the secret to runtime and then passes it to 
DbtShellTask
c

Constantino Schillebeeckx

08/07/2021, 12:03 AM
Ah excellent, thank you @emre that did indeed do the trick!
View count: 7
Back to #prefect-communityJoin thread in Slack