Philip MacMenamin
04/14/2022, 6:50 PM2022-04-14 09:42:06-0600] ERROR - prefect.TaskRunner | Task 'ShellTask[0]': Exception encountered during task execution!
Traceback (most recent call last):
File "/blah/python3.9/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/blah/python3.9/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/blah/python3.9/site-packages/prefect/utilities/tasks.py", line 456, in method
return run_method(self, *args, **kwargs)
File "/blah/python3.9/site-packages/prefect/tasks/shell.py", line 131, in run
tmp.write(command.encode())
AttributeError: 'list' object has no attribute 'encode'
I have a couple of questions:
• Is there a way to tag shell_tasks such that you can see some clue as to which one failed?
• Can I get a better description of the failure
At the moment I have
shell_task = ShellTask(log_stderr=True, return_all=True, stream_output=True)
Kevin Kho
04/14/2022, 6:52 PMPhilip MacMenamin
04/14/2022, 6:55 PMshell_task
and I'm throwing different commands at it. Ideally I'd like to keep that, and just give it diff labels.Kevin Kho
04/14/2022, 6:57 PMPhilip MacMenamin
04/14/2022, 7:04 PMKevin Kho
04/14/2022, 7:07 PMPhilip MacMenamin
04/14/2022, 10:31 PMtask_run_name
or something along those lines?Kevin Kho
04/14/2022, 10:45 PMtask_run_name
to the ShellTask also. The task library should take the standard task kwargs otherwise it’s a bugShellTask(…,task_run_name=….)
Philip MacMenamin
04/15/2022, 1:56 PMShellTask
object with the task_run_name={val}
but then I can't load anything into var
because run()
doesn't take kwargs. Or no?task = ShellTask(task_run_name="{val}")
with Flow("My Flow") as f:
contents = task(command='ls', task_run_name='list task')
things = task(command='other command', task_run_name='other task')
Kevin Kho
04/15/2022, 1:58 PMval
there. But you can make your own copy of the ShellTask that just takes the val
and doesn’t do anythingPhilip MacMenamin
04/15/2022, 2:03 PMtask_list = ShellTask(task_run_name="does listing")
task_thing = ShellTask(task_run_name="does thing")
?
As in create a shell task object per type of command? The idea was I didn't want to have to create another ShellTask object per command, I just wanted to give them diff labels / some way I could ID them.Kevin Kho
04/15/2022, 2:05 PMfrom prefect.tasks.shell import ShellTask
from typing import Any, List, Union, Optional
import prefect
from prefect.utilities.tasks import defaults_from_attrs
def echo(command):
return f"echo {command}"
class MyShellTask(ShellTask):
@defaults_from_attrs("command", "env", "helper_script")
def run(
self, command: str = None, env: dict = None, helper_script: str = None, new_arg: str=None
) -> Optional[Union[str, List[str]]]:
new_command = echo(command) + new_arg
super().run(command=new_command, env=env, helper_script=helper_script)
MyShellTask(log_stdout=True, stream_output=True, task_run_name="{new_arg}").run(command="1", new_arg="test")
Philip MacMenamin
04/15/2022, 2:17 PM