Thread
#prefect-community
    Madison Schott

    Madison Schott

    7 months ago
    Hi all, I'm looking to run a command within my Docker container in production in order to generate a slack notification for a dbt open-source package. Is there a specific type of task I would need to write for this within my Python file? The command is as follows:
    re_data notify slack \
    --start-date 2021-01-01 \
    --end-date 2021-01-31 \
    --webhook-url <https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX> \
    --subtitle="[Optional] Markdown text to be added as a subtitle in the slack message generated"
    Could I do something like this?
    from prefect.tasks.shell import ShellTask
    
    task = ShellTask(helper_script="cd ~")
    re_data_slack = task(command='re_data notify slack')
    Also wondering how I would download the dbt package onto the Docker container that my flow is running on
    Probably just
    RUN pip install re_data
    in the Dockerfile I'm using?
    Kevin Kho

    Kevin Kho

    7 months ago
    Yes i think the ShellTask is the way to do it. Yes also to the install.
    Or I guess
    pip install re-data
    if it’s this . you might be using the underscore
    Madison Schott

    Madison Schott

    7 months ago
    hmm the shell task failed but not sure why
    from prefect.tasks.shell import ShellTask
    
    shell_task = ShellTask(helper_script="cd ~")
    
    re_data_slack_alert = shell_task(command='re_data notify slack \
                                                    --start-date 2022-01-01 \
                                                    --end-date 2022-07-31 \
                                                    --webhook-url xxx')
    Kevin Kho

    Kevin Kho

    7 months ago
    Can you try adding
    log_stderr=True
    and
    return_all=True
    and
    stream_output=True
    to the shell task so we can try to get more logs?>
    Exit code 1 is operation not permitted though, does this work in your execution environment without Prefect?
    Madison Schott

    Madison Schott

    7 months ago
    yeah it does
    Kevin Kho

    Kevin Kho

    7 months ago
    Does adding the above stuff give you more logs?
    Madison Schott

    Madison Schott

    7 months ago
    added the same helper script that my dbtShellTask is using
    shell_task = ShellTask(helper_script="cd /dbt_snowflake/", log_stderr=True, return_all=True, stream_output=True)
    does this need to be in a dbtShellTask??
    Kevin Kho

    Kevin Kho

    7 months ago
    The dbt shell task is really just a subclass of the shell task with additional arguments so if you reconstruct the exact same CLI in ShellTask, it would be equivalent. The dbtshelltask can set up some things that might help so it’s worth a shot. For this specifically, I have never seen this re_data, but I guess based on the errors you need to find a way to get a profile detected right? I could see the DBTShellTask helping with that but I’m not sure
    Madison Schott

    Madison Schott

    7 months ago
    Still having some issues, now it's having trouble finding a macro in a package that I downloaded
    But I see the macro listed there under that package and I have that package in my
    packages.yml
    Any ideas? Would I need to run
    dbt deps
    ?
    Kevin Kho

    Kevin Kho

    7 months ago
    Let me find someone who knows dbt more
    Madison Schott

    Madison Schott

    7 months ago
    So apparently this was added as the latest release, which I did update in my packages.yml- I would think this new version would copy onto my Docker environment but maybe not? I'm not sure
    Step 6/14 : RUN pip install re_data
     ---> Running in 8a00459332d9
    Collecting re_data
      Downloading re_data-0.6.0-py3-none-any.whl (543 kB)
    Looks like is downloaded the right version
    Kevin Kho

    Kevin Kho

    7 months ago
    What is your RunConfig?
    Madison Schott

    Madison Schott

    7 months ago
    RUN_CONFIG = ECSRun(run_task_kwargs={'cluster': 'prefect-prod'},
                        env={"PREFECT__LOGGING__LEVEL": "DEBUG"},
                        execution_role_arn='arn:aws:iam::xx',
                        labels=['ecs-agent', 'prod', 'winc'])
    Kevin Kho

    Kevin Kho

    7 months ago
    That should use the latest image successfully I think
    Madison Schott

    Madison Schott

    7 months ago
    hmm still saying it can't find the macro in the package, which came with the latest updates
    Kevin Kho

    Kevin Kho

    7 months ago
    Noone is familiar with this on the team. Are you going into the container and seeing that macro?