Constantino Schillebeeckx
08/06/2021, 5:56 PMDbtShellTask
along with secrets (more details in thread)Task 'DbtShellTask': Exception encountered during task execution!
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 865, in get_task_run_state
logger=self.logger,
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 328, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 441, in method
return run_method(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/dbt/dbt.py", line 183, in run
command=command, env=env, helper_script=helper_script
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 441, in method
return run_method(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/shell.py", line 140, in run
shell=sys.platform == "win32",
File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.7/subprocess.py", line 1462, in _execute_child
env_list.append(k + b'=' + os.fsencode(v))
File "/usr/local/lib/python3.7/os.py", line 812, in fsencode
filename = fspath(filename) # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not Secret
Failed to load and execute Flow's environment: ValueError('Secrets should only be retrieved during a Flow run, not while building a Flow.')
DbtShellTask
env with secrets? cc @Kevin KhoTask 'DbtShellTask': Exception encountered during task execution!
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 865, in get_task_run_state
logger=self.logger,
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 328, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 441, in method
return run_method(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/dbt/dbt.py", line 183, in run
command=command, env=env, helper_script=helper_script
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 441, in method
return run_method(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/shell.py", line 140, in run
shell=sys.platform == "win32",
File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.7/subprocess.py", line 1462, in _execute_child
env_list.append(k + b'=' + os.fsencode(v))
File "/usr/local/lib/python3.7/os.py", line 812, in fsencode
filename = fspath(filename) # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not PrefectSecret
emre
08/06/2021, 7:21 PM__init__
time, while they are available in run
time.
Your last example uses PrefectSecret
tasks to fetch credentials. That is a Task
and therefore it will fetch the credentials when you do flow.run()
, not within with Flow():
. However, you are passing these tasks to DbtShellTask.__init__
under the env
parameter.env
parameter from the first set of parentheses (__init__
) toı the second one, which will correctly set the credentials as an upstream dependency for the runtime.
DbtShellTask(profiles_dir='', ...)(
command='dbt run',
env = {
'USERNAME': PrefectSecret('User'),
'PASSWORD': PrefectSecret('Pass')
}
)
Kevin Kho
08/06/2021, 7:46 PMSecret("xxx").get()
to get it in the first example.
Emre’s example seems like the easiest approach. It defers loading the secret to runtime and then passes it to DbtShellTask
Constantino Schillebeeckx
08/07/2021, 12:03 AM