https://prefect.io logo
Title
s

Scott Henley

07/08/2022, 6:30 PM
Greetings, I am having an issue registering a Flow with a simple ShellTask (Prefect/Cloud 1.0). The RunConfig for the flow is pointing to a custom image in ECR that contains the binary that we are attempting to run via the shell task (using the ECSRun type). The path to the binary in the shell command does not exist locally. When I attempt to register the flow, it fails with a “no such file or directory” error. Does the register cli command attempt to verify the shell task locally? How can I get around this when I only want it to reference the location in the container image?
k

Kevin Kho

07/08/2022, 6:32 PM
It should need it. Can I see how you call the ShellTask?
s

Scott Henley

07/08/2022, 6:36 PM
I create the task:
dcm_task = ShellTask(
    return_all=True,
    stream_output=True
)
and run it in the flow:
output = dcm_task.run(command='./service/dcm-service')
where the path to the binary inside the ECR image is the command
I mainly want to get back the log data from the binary service and output it.
k

Kevin Kho

07/08/2022, 6:39 PM
Can I see your traceback?
Ah wait. don’t call run in the flow
Just do:
output = dcm_task.run(command='./service/dcm-service')
s

Scott Henley

07/08/2022, 6:40 PM
When I use the register cli command this is the error:
Collecting flows...
[2022-07-07 15:12:55-0500] INFO - prefect.ShellTask | /var/folders/7d/41lxpm4s6039384cprdz33rm0000gn/T/prefect-va5kr_fc: line 1: /service/dcm-service: No such file or directory
[2022-07-07 15:12:55-0500] ERROR - prefect.ShellTask | Command failed with exit code 127
Traceback (most recent call last):
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/click/core.py", line 1635, in invoke
    rv = super().invoke(ctx)
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/prefect/cli/build_register.py", line 64, in inner
    return func(*args, **kwargs)
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/prefect/cli/build_register.py", line 810, in register
    register_internal(
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/prefect/cli/build_register.py", line 536, in register_internal
    source_to_flows = collect_flows(
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/prefect/cli/build_register.py", line 255, in collect_flows
    flows = load_flows_from_script(s.location)
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/prefect/cli/build_register.py", line 133, in load_flows_from_script
    namespace = runpy.run_path(abs_path, run_name="<flow>")
  File "/Users/scotthenley/.pyenv/versions/3.9.12/lib/python3.9/runpy.py", line 268, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Users/scotthenley/.pyenv/versions/3.9.12/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/scotthenley/.pyenv/versions/3.9.12/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/scotthenley/Projects/prefect-flows/prefect/test/dcm_api_service.py", line 36, in <module>
    output = dcm_task.run(command='/service/dcm-service')
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/prefect/utilities/tasks.py", line 456, in method
    return run_method(self, *args, **kwargs)
  File "/Users/scotthenley/Projects/prefect-flows/.direnv/python-3.9.12/lib/python3.9/site-packages/prefect/tasks/shell.py", line 164, in run
    raise prefect.engine.signals.FAIL(
prefect.engine.signals.FAIL: Command failed with exit code 127
k

Kevin Kho

07/08/2022, 6:40 PM
You shouldn’t be calling .run in the Flow. The Flow block will do that for you
s

Scott Henley

07/08/2022, 6:41 PM
ahh, so how do I reference the task in the flow block?
k

Kevin Kho

07/08/2022, 6:41 PM
shell = ShellTask(..)
with Flow(..) as flow:
    shell(...)
s

Scott Henley

07/08/2022, 6:42 PM
can insert the command in the flow block like this:
output = dcm_task(command='/service/dcm-service')
or should command be part of the ShellTask initialization?
k

Kevin Kho

07/08/2022, 6:44 PM
It can be inserted in both places
s

Scott Henley

07/08/2022, 6:45 PM
awesome thanks, that did the trick
it makes sense.. run() tries to run it immediately at registration
👀 1
k

Kevin Kho

07/08/2022, 6:46 PM
Yes exactly!