https://prefect.io
Join Slack
I'm looking through the `ShellTask` <source> because I was trying to see if there's a way to stream ...
# ask-community
s
I'm looking through the 
ShellTask
source because I was trying to see if there's a way to stream the task output when the log level is 
INFO
. Is it possible? I have a task whose output i'd like to regularly stream to my logs, but I don't want to have the entire flow running in debug mode
c
I was doing the exact same thing like two weeks ago - my solution was to copy the task class and change the logger line to info
Not a great solution, but I did’t see any other way
s
ooooof
for what i'm doing i think i would rather just forgo the streaming entirely....
alright well good to know someone else was trying to do the same
c
yeah, might be worth a feature request :)
upvote 1
💯 1
d
Hi @Sean Talia! I’m not sure I totally understand what you’re going for — are you attempting to change the logging level of a specific Task?
s
yep that's exactly what i'm trying to do
I have several shell tasks (specifically `DbtShellTask`s) that I'm executing in a flow and there are a couple of those shell commands that normally log to stdout
d
Gotcha
Definitely open an issue for an enhancement!
And you can can always contribute an improvement to the 
ShellTask
that allows you to more directly configure the logging level. We’d welcome your PR!
👍 1
c
Funny, I was also looking that that task because I’ running DBT through it 😉
s
I was trying to solve this by adding the dbt packages to the list of additional loggers but wasn't having luck there; i assume the issue is the fact that dbt is being interfaced with here through a subprocess rather than directly through prefect
✅ 1
c
yep, it’s a separate python process
☝️ 1
👍 1
I was debating if I should call the dbt functions directly instead of using a shell task, but the dbt docs explicitly discourage that
😱 1
So in the end I ditched the DbtShell task (it didn’t add much value in my setup), copied the ShellTask with the change to the log level and used that to execute dbt - works well enough
s
alright yeah if no one else gets around to it in the next couple of weeks I'll probably end up submitting a PR when I have some time
i'm sure you and I will not be the only ones who want/need this @Christoph Wiese 😎
👍 2
c
Just adding my +1 here if you guys plan on working on this 🙂. I’m running into this same scenario now. Setting flow logs to 
DEBUG
seems to be the only way to get the output of 
DbtShellTask
to display in cloud.
s
so, I'm working on a PR for this right now; I think that I actually have most of what would be needed to make this happen, but I'm still working on actually locally installing this version of prefect that I have sitting up on this branch of my forked repo and testing with the 
DbtShellTask
sadly i'm dealing with some 
dbt
issues right now that are making it difficult to install in my local virtualenv and i don't have time to tend to this atm; @Christoph Wiese @Chris Vrooman watch out for this: https://github.com/fishtown-analytics/dbt/issues/3160
c
thanks for the heads up! I had the same issue with 0.19 the other day, but poetry. was also showing me a message that my pip was out of date and updating this fixed it as suggested in the ticket - now I know the root cause for that whole issue!
s
alright @Christoph Wiese @Chris Vrooman jim just merged my PR so hopefully this will go out in the next release?
🙌🏻 1
upvote 2
d
@Sean Talia it’s available right now in 
master
! We also release every Tuesday 😄
❤️ 1
s
awesome! super exciting
c
thank you @Sean Talia, I’ll pick this up in my internal release today!
s
ermmm @Dylan did I mess something up here with the raises block in the docstring? this is not rendering so hot in the api docs: https://docs.prefect.io/api/latest/tasks/shell.html#shelltask
d
Hey @Sean Talia can you send a new message to #C0106HZ1CMS?
s
yep no problem
d
Thanks!
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by