Constantino Schillebeeckx
08/06/2021, 5:56 PMDbtShellTask
along with secrets (more details in thread)Constantino Schillebeeckx
08/06/2021, 5:57 PMTask 'DbtShellTask': Exception encountered during task execution!
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 865, in get_task_run_state
logger=self.logger,
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 328, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 441, in method
return run_method(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/dbt/dbt.py", line 183, in run
command=command, env=env, helper_script=helper_script
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 441, in method
return run_method(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/shell.py", line 140, in run
shell=sys.platform == "win32",
File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.7/subprocess.py", line 1462, in _execute_child
env_list.append(k + b'=' + os.fsencode(v))
File "/usr/local/lib/python3.7/os.py", line 812, in fsencode
filename = fspath(filename) # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not Secret
Constantino Schillebeeckx
08/06/2021, 5:58 PMFailed to load and execute Flow's environment: ValueError('Secrets should only be retrieved during a Flow run, not while building a Flow.')
Constantino Schillebeeckx
08/06/2021, 5:58 PMDbtShellTask
env with secrets? cc @Kevin KhoConstantino Schillebeeckx
08/06/2021, 6:52 PMTask 'DbtShellTask': Exception encountered during task execution!
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 865, in get_task_run_state
logger=self.logger,
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 328, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 441, in method
return run_method(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/dbt/dbt.py", line 183, in run
command=command, env=env, helper_script=helper_script
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 441, in method
return run_method(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/shell.py", line 140, in run
shell=sys.platform == "win32",
File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.7/subprocess.py", line 1462, in _execute_child
env_list.append(k + b'=' + os.fsencode(v))
File "/usr/local/lib/python3.7/os.py", line 812, in fsencode
filename = fspath(filename) # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not PrefectSecret
emre
08/06/2021, 7:21 PM__init__
time, while they are available in run
time.
Your last example uses PrefectSecret
tasks to fetch credentials. That is a Task
and therefore it will fetch the credentials when you do flow.run()
, not within with Flow():
. However, you are passing these tasks to DbtShellTask.__init__
under the env
parameter.emre
08/06/2021, 7:26 PMenv
parameter from the first set of parentheses (__init__
) toı the second one, which will correctly set the credentials as an upstream dependency for the runtime.
DbtShellTask(profiles_dir='', ...)(
command='dbt run',
env = {
'USERNAME': PrefectSecret('User'),
'PASSWORD': PrefectSecret('Pass')
}
)
Kevin Kho
Secret("xxx").get()
to get it in the first example.
Emre’s example seems like the easiest approach. It defers loading the secret to runtime and then passes it to DbtShellTask
Constantino Schillebeeckx
08/07/2021, 12:03 AM