https://prefect.io
Join Slack
We use prefect 2 and all of our flows use `prefect-shell` to send commands to spin up docker contain...
# prefect-getting-started
s
We use prefect 2 and all of our flows use 
prefect-shell
to send commands to spin up docker containers. We do get some logs from within the container processes but not all of them - then we have to find the container or process on our VM and look at the logs there - it’d be great to have all logs show up in the prefect dashboard. It’s not clear to me why we get some logs and not others from our docker containers. Any guidance would be appreciated.
👀 1
a
I have a refactored block here which also handles stderr logs better I think: https://github.com/PrefectHQ/prefect-shell/pull/55
s
Thanks. We just use 
shell_run_command
directly, not as a block. Do the changes only affect use as a block?
a
yes, the changes only affect use as a block 
ShellOperation
does a complete overhaul of the implementation; particularly, I think it should better handle stderr. Previously stderr was only used to catch exceptions.
s
Okay, but it sounds like I could switch from using 
shell_run_command
to using 
ShellOperation
and then hopefully improve our logs
a
yeah it might help! I’ll try to get all the PRs merged soon and do a release. at the moment, ShellOperation is only available on the main branch
s
Great. Thanks
I see there’s a new version of 
prefect-shell
with 
ShellOperation
- looks like it can be used as a block or not. what are the main reasons to use as a block or not?
a
I think it can only be used as a block if I’m not mistaken. but I believe we would want a flow alongside it soon
s
Okay
a
curious where did you see it used not as a block?
s
It’s a little hard to wrap my head around, as at least 
shell_run_command
was a task essentially, and i don’t see anything in the docs about tasks as blocks.
a
oh that’s still a block [class] (just not loaded)
s
oh ok
a
if you want, you can use these blocks outside flows / tasks, as shown in the example
s
no, i want to use inside tasks/flows, just trying to figure out how to translate my use of 
shell_run_command
to 
ShellOperation
a
Copy code
@task
def run_commands(commands, env=env):
    return ShellOperation(commands=commands, env=env).run()
or
Copy code
@task
def run_commands(commands, env=env):
    op = ShellOperation(commands=commands, env=env)
    process = op.trigger()
    # do something else here

    process.wait_for_completion()
    return process.fetch_result()
s
Great, thanks
a
Theoretically, wait_for_completion and fetch_result can be in their own tasks too
s
okay.
Curious how you recommend using 
ShellOperation
inside reusable tasks used in many different flows?
a
save it once and load 
ShellOperation.load
s
Okay
Right i see tht now
👍 1
a
Copy code
@flow
def run_commands(commands, env=env):
    return ShellOperation.load("BLOCK_NAME").run()
s
Thanks
Basically all our tasks are shell commands. We’ve got hundreds of them where each is a different command. It looks like you save a block that has a specific command. Would we save a block for each unique command we have? Each of which are tasks probably
a
depends if you can link the commands; one shelloperation block accepts multiple commands
🤔 1
like the example on the homepage (cd, mkdir)
s
Okay, i’ll play around to see what works
👍 1
a
let me know if you encounter an issue, and thanks for helping us test!
s
Hmm, tried
Copy code
from prefect_shell import ShellOperation
sleep_op = ShellOperation(commands=["sleep 5", "echo 'Hello, world!'"])
sleep_op.save("sleep-op-block2")
and I got a uuid 
UUID('733f8c33-aa2c-486e-84f8-5e382c67463f')
but it didn’t save to the Prefect UI
I can go to the Prefect UI and add a shell operation block, but doing it via code above didn’t work
and I did run 
prefect block register -m prefect_shell
a
weird I see it: ``````
s
hmm
ah okay, sorry
I needed to use 
sudo ipython
on my VM
👀 1
to pick up credentials
a
I see. no worries
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by