https://prefect.io logo
Powered by
Title
s

scott

02/01/2023, 8:10 PM
We use prefect 2 and all of our flows use 
prefect-shell
to send commands to spin up docker containers. We do get some logs from within the container processes but not all of them - then we have to find the container or process on our VM and look at the logs there - it’d be great to have all logs show up in the prefect dashboard. It’s not clear to me why we get some logs and not others from our docker containers. Any guidance would be appreciated.
👀 1
a

Andrew Huang

02/01/2023, 8:25 PM
I have a refactored block here which also handles stderr logs better I think: https://github.com/PrefectHQ/prefect-shell/pull/55
s

scott

02/01/2023, 8:36 PM
Thanks. We just use 
shell_run_command
directly, not as a block. Do the changes only affect use as a block?
a

Andrew Huang

02/01/2023, 8:38 PM
yes, the changes only affect use as a block 
ShellOperation
does a complete overhaul of the implementation; particularly, I think it should better handle stderr. Previously stderr was only used to catch exceptions.
s

scott

02/01/2023, 8:40 PM
Okay, but it sounds like I could switch from using 
shell_run_command
to using 
ShellOperation
and then hopefully improve our logs
a

Andrew Huang

02/01/2023, 8:40 PM
yeah it might help! I’ll try to get all the PRs merged soon and do a release. at the moment, ShellOperation is only available on the main branch
s

scott

02/01/2023, 8:42 PM
Great. Thanks
I see there’s a new version of 
prefect-shell
with 
ShellOperation
- looks like it can be used as a block or not. what are the main reasons to use as a block or not?
a

Andrew Huang

02/02/2023, 10:35 PM
I think it can only be used as a block if I’m not mistaken. but I believe we would want a flow alongside it soon
s

scott

02/02/2023, 10:35 PM
Okay
a

Andrew Huang

02/02/2023, 10:35 PM
curious where did you see it used not as a block?
s

scott

02/02/2023, 10:36 PM
https://prefecthq.github.io/prefect-shell/examples_catalog/
It’s a little hard to wrap my head around, as at least 
shell_run_command
was a task essentially, and i don’t see anything in the docs about tasks as blocks.
a

Andrew Huang

02/02/2023, 10:36 PM
oh that’s still a block [class] (just not loaded)
s

scott

02/02/2023, 10:36 PM
oh ok
a

Andrew Huang

02/02/2023, 10:37 PM
if you want, you can use these blocks outside flows / tasks, as shown in the example
s

scott

02/02/2023, 10:38 PM
no, i want to use inside tasks/flows, just trying to figure out how to translate my use of 
shell_run_command
to 
ShellOperation
a

Andrew Huang

02/02/2023, 10:41 PM
@task
def run_commands(commands, env=env):
    return ShellOperation(commands=commands, env=env).run()
or 
@task
def run_commands(commands, env=env):
    op = ShellOperation(commands=commands, env=env)
    process = op.trigger()
    # do something else here

    process.wait_for_completion()
    return process.fetch_result()
s

scott

02/02/2023, 10:42 PM
Great, thanks
a

Andrew Huang

02/02/2023, 10:42 PM
Theoretically, wait_for_completion and fetch_result can be in their own tasks too
s

scott

02/02/2023, 10:42 PM
okay.
Curious how you recommend using 
ShellOperation
inside reusable tasks used in many different flows?
a

Andrew Huang

02/02/2023, 10:43 PM
save it once and load 
ShellOperation.load
image.png
s

scott

02/02/2023, 10:44 PM
Okay
Right i see tht now
👍 1
a

Andrew Huang

02/02/2023, 10:44 PM
@flow
def run_commands(commands, env=env):
    return ShellOperation.load("BLOCK_NAME").run()
s

scott

02/02/2023, 10:45 PM
Thanks
Basically all our tasks are shell commands. We’ve got hundreds of them where each is a different command. It looks like you save a block that has a specific command. Would we save a block for each unique command we have? Each of which are tasks probably
a

Andrew Huang

02/02/2023, 10:45 PM
depends if you can link the commands; one shelloperation block accepts multiple commands
🤔 1
like the example on the homepage (cd, mkdir)
s

scott

02/02/2023, 10:49 PM
Okay, i’ll play around to see what works
👍 1
a

Andrew Huang

02/02/2023, 10:54 PM
let me know if you encounter an issue, and thanks for helping us test!
s

scott

02/02/2023, 11:04 PM
Hmm, tried 
from prefect_shell import ShellOperation
sleep_op = ShellOperation(commands=["sleep 5", "echo 'Hello, world!'"])
sleep_op.save("sleep-op-block2")
and I got a uuid 
UUID('733f8c33-aa2c-486e-84f8-5e382c67463f')
but it didn’t save to the Prefect UI
I can go to the Prefect UI and add a shell operation block, but doing it via code above didn’t work
and I did run 
prefect block register -m prefect_shell
a

Andrew Huang

02/02/2023, 11:11 PM
weird I see it: ``````
s

scott

02/02/2023, 11:12 PM
hmm
ah okay, sorry
I needed to use 
sudo ipython
on my VM
👀 1
to pick up credentials
a

Andrew Huang

02/02/2023, 11:14 PM
I see. no worries
3 Views
#prefect-getting-startedJoin Slack
Powered by