https://prefect.io
Join Slack
Hi all, I'm looking to run a command within my Docker container in production in order to generate a...
# ask-community
m
Hi all, I'm looking to run a command within my Docker container in production in order to generate a slack notification for a dbt open-source package. Is there a specific type of task I would need to write for this within my Python file? The command is as follows:
Copy code
re_data notify slack \
--start-date 2021-01-01 \
--end-date 2021-01-31 \
--webhook-url <https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX> \
--subtitle="[Optional] Markdown text to be added as a subtitle in the slack message generated"
Could I do something like this?
Copy code
from prefect.tasks.shell import ShellTask

task = ShellTask(helper_script="cd ~")
re_data_slack = task(command='re_data notify slack')
Also wondering how I would download the dbt package onto the Docker container that my flow is running on
Probably just 
RUN pip install re_data
in the Dockerfile I'm using?
k
Yes i think the ShellTask is the way to do it. Yes also to the install.
Or I guess 
pip install re-data
if it’s this . you might be using the underscore
m
hmm the shell task failed but not sure why
Copy code
from prefect.tasks.shell import ShellTask

shell_task = ShellTask(helper_script="cd ~")

re_data_slack_alert = shell_task(command='re_data notify slack \
                                                --start-date 2022-01-01 \
                                                --end-date 2022-07-31 \
                                                --webhook-url xxx')
k
Can you try adding 
log_stderr=True
and 
return_all=True
and 
stream_output=True
to the shell task so we can try to get more logs?>
Exit code 1 is operation not permitted though, does this work in your execution environment without Prefect?
m
yeah it does
k
Does adding the above stuff give you more logs?
m
added the same helper script that my dbtShellTask is using 
shell_task = ShellTask(helper_script="cd /dbt_snowflake/", log_stderr=True, return_all=True, stream_output=True)
does this need to be in a dbtShellTask??
k
The dbt shell task is really just a subclass of the shell task with additional arguments so if you reconstruct the exact same CLI in ShellTask, it would be equivalent. The dbtshelltask can set up some things that might help so it’s worth a shot. For this specifically, I have never seen this re_data, but I guess based on the errors you need to find a way to get a profile detected right? I could see the DBTShellTask helping with that but I’m not sure
m
Still having some issues, now it's having trouble finding a macro in a package that I downloaded
But I see the macro listed there under that package and I have that package in my 
packages.yml
Any ideas? Would I need to run 
dbt deps
?
k
Let me find someone who knows dbt more
m
So apparently this was added as the latest release, which I did update in my packages.yml- I would think this new version would copy onto my Docker environment but maybe not? I'm not sure
Copy code
Step 6/14 : RUN pip install re_data
 ---> Running in 8a00459332d9
Collecting re_data
  Downloading re_data-0.6.0-py3-none-any.whl (543 kB)
Looks like is downloaded the right version
k
What is your RunConfig?
m
Copy code
RUN_CONFIG = ECSRun(run_task_kwargs={'cluster': 'prefect-prod'},
                    env={"PREFECT__LOGGING__LEVEL": "DEBUG"},
                    execution_role_arn='arn:aws:iam::xx',
                    labels=['ecs-agent', 'prod', 'winc'])
k
That should use the latest image successfully I think
m
hmm still saying it can't find the macro in the package, which came with the latest updates
k
Noone is familiar with this on the team. Are you going into the container and seeing that macro?
8 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by