scott
02/01/2023, 8:10 PMprefect-shell
to send commands to spin up docker containers. We do get some logs from within the container processes but not all of them - then we have to find the container or process on our VM and look at the logs there - it’d be great to have all logs show up in the prefect dashboard. It’s not clear to me why we get some logs and not others from our docker containers. Any guidance would be appreciated.Andrew Huang
02/01/2023, 8:25 PMscott
02/01/2023, 8:36 PMshell_run_command
directly, not as a block. Do the changes only affect use as a block?Andrew Huang
02/01/2023, 8:38 PMShellOperation
does a complete overhaul of the implementation; particularly, I think it should better handle stderr. Previously stderr was only used to catch exceptions.scott
02/01/2023, 8:40 PMshell_run_command
to using ShellOperation
and then hopefully improve our logsAndrew Huang
02/01/2023, 8:40 PMscott
02/01/2023, 8:42 PMprefect-shell
with ShellOperation
- looks like it can be used as a block or not. what are the main reasons to use as a block or not?Andrew Huang
02/02/2023, 10:35 PMscott
02/02/2023, 10:35 PMAndrew Huang
02/02/2023, 10:35 PMscott
02/02/2023, 10:36 PMshell_run_command
was a task essentially, and i don’t see anything in the docs about tasks as blocks.Andrew Huang
02/02/2023, 10:36 PMscott
02/02/2023, 10:36 PMAndrew Huang
02/02/2023, 10:37 PMscott
02/02/2023, 10:38 PMshell_run_command
to ShellOperation
Andrew Huang
02/02/2023, 10:41 PM@task
def run_commands(commands, env=env):
return ShellOperation(commands=commands, env=env).run()
or
@task
def run_commands(commands, env=env):
op = ShellOperation(commands=commands, env=env)
process = op.trigger()
# do something else here
process.wait_for_completion()
return process.fetch_result()
scott
02/02/2023, 10:42 PMAndrew Huang
02/02/2023, 10:42 PMscott
02/02/2023, 10:42 PMShellOperation
inside reusable tasks used in many different flows?Andrew Huang
02/02/2023, 10:43 PMShellOperation.load
scott
02/02/2023, 10:44 PMAndrew Huang
02/02/2023, 10:44 PM@flow
def run_commands(commands, env=env):
return ShellOperation.load("BLOCK_NAME").run()
scott
02/02/2023, 10:45 PMAndrew Huang
02/02/2023, 10:45 PMscott
02/02/2023, 10:49 PMAndrew Huang
02/02/2023, 10:54 PMscott
02/02/2023, 11:04 PMfrom prefect_shell import ShellOperation
sleep_op = ShellOperation(commands=["sleep 5", "echo 'Hello, world!'"])
sleep_op.save("sleep-op-block2")
and I got a uuid UUID('733f8c33-aa2c-486e-84f8-5e382c67463f')
but it didn’t save to the Prefect UIprefect block register -m prefect_shell
Andrew Huang
02/02/2023, 11:11 PMscott
02/02/2023, 11:12 PMsudo ipython
on my VMAndrew Huang
02/02/2023, 11:14 PM