Channels
#prefect-community
Title
m
Madison Schott
02/08/2022, 8:19 PMHi all, I'm looking to run a command within my Docker container in production in order to generate a slack notification for a dbt open-source package. Is there a specific type of task I would need to write for this within my Python file? The command is as follows:
Copy code
re_data notify slack \
--start-date 2021-01-01 \
--end-date 2021-01-31 \
--webhook-url <https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX> \
--subtitle="[Optional] Markdown text to be added as a subtitle in the slack message generated"Could I do something like this?
Copy code
from prefect.tasks.shell import ShellTask
task = ShellTask(helper_script="cd ~")
re_data_slack = task(command='re_data notify slack')Also wondering how I would download the dbt package onto the Docker container that my flow is running on
Probably just
RUN pip install re_datak
Kevin Kho
02/08/2022, 8:32 PM
Yes i think the ShellTask is the way to do it. Yes also to the install.
Or I guess
pip install re-datam
Madison Schott
02/09/2022, 6:05 PMhmm the shell task failed but not sure why
Copy code
from prefect.tasks.shell import ShellTask
shell_task = ShellTask(helper_script="cd ~")
re_data_slack_alert = shell_task(command='re_data notify slack \
--start-date 2022-01-01 \
--end-date 2022-07-31 \
--webhook-url xxx')k
Kevin Kho
02/09/2022, 6:31 PM
Can you try adding
log_stderr=Truereturn_all=Truestream_output=TrueExit code 1 is operation not permitted though, does this work in your execution environment without Prefect?
m
Madison Schott
02/09/2022, 8:09 PMyeah it does
k
Kevin Kho
02/09/2022, 8:26 PM
Does adding the above stuff give you more logs?
m
Madison Schott
02/10/2022, 10:31 PMadded the same helper script that my dbtShellTask is using
shell_task = ShellTask(helper_script="cd /dbt_snowflake/", log_stderr=True, return_all=True, stream_output=True)does this need to be in a dbtShellTask??
k
Kevin Kho
02/10/2022, 10:40 PM
The dbt shell task is really just a subclass of the shell task with additional arguments so if you reconstruct the exact same CLI in ShellTask, it would be equivalent.
The dbtshelltask can set up some things that might help so it’s worth a shot.
For this specifically, I have never seen this re_data, but I guess based on the errors you need to find a way to get a profile detected right? I could see the DBTShellTask helping with that but I’m not sure
m
Madison Schott
02/11/2022, 4:53 PMStill having some issues, now it's having trouble finding a macro in a package that I downloaded
But I see the macro listed there under that package and I have that package in my
packages.ymlAny ideas? Would I need to run
dbt depsk
Kevin Kho
02/11/2022, 5:16 PM
Let me find someone who knows dbt more
m
Madison Schott
02/11/2022, 5:29 PMSo apparently this was added as the latest release, which I did update in my packages.yml- I would think this new version would copy onto my Docker environment but maybe not? I'm not sure
Copy code
Step 6/14 : RUN pip install re_data
---> Running in 8a00459332d9
Collecting re_data
Downloading re_data-0.6.0-py3-none-any.whl (543 kB)Looks like is downloaded the right version
k
Kevin Kho
02/11/2022, 5:38 PM
What is your RunConfig?
m
Madison Schott
02/11/2022, 5:39 PM Copy code
RUN_CONFIG = ECSRun(run_task_kwargs={'cluster': 'prefect-prod'},
env={"PREFECT__LOGGING__LEVEL": "DEBUG"},
execution_role_arn='arn:aws:iam::xx',
labels=['ecs-agent', 'prod', 'winc'])k
Kevin Kho
02/11/2022, 5:40 PM
That should use the latest image successfully I think
m
Madison Schott
02/11/2022, 6:30 PMhmm still saying it can't find the macro in the package, which came with the latest updates
k
Kevin Kho
02/11/2022, 8:49 PM
Noone is familiar with this on the team. Are you going into the container and seeing that macro?
5 Views