Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi! I'm standing up dbt CLI on AWS Fargate using tasks from this template:
<https://github.com/anna-geller/dataflow-ops>

I have all of the flows deployed and scheduled using Prefect cloud, and I'm having no problems running the dbt run (it is using the project files from the S3 bucket/folder that's attached to the image), but I'm running into issues when running `dbt docs generate` as the Flow Run is outputting that
`Catalog written to /opt/prefect/target/catalog.json`

I checked S3 and it's definitely not there, and I'm guessing that the `/opt` is a directory that's tied to the ECS Task from the container? Is there a way that the dbt cli command can output the docs file (specifically index.html into the S3 bucket so we can host our dbt docs as a static site)

Flow code is in the :thread:. I really appreciate any help or pointers :slightly_smiling_face:

```from prefect import task, flow
from prefect import get_run_logger

from prefect import flow
from prefect_dbt.cli.commands import trigger_dbt_cli_command

from prefect_snowflake.credentials import SnowflakeCredentials
from prefect_snowflake.database import SnowflakeConnector

from prefect_dbt.cli.commands import trigger_dbt_cli_command
from prefect_dbt.cli.configs import SnowflakeTargetConfigs
from prefect_dbt.cli import DbtCliProfile
from prefect.blocks.system import Secret

@flow
def build_dbt_docs():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Calling dbt deps")
    
    # Access the stored secret
    secret_block = Secret.load("XXX")

    connector = SnowflakeConnector(
        schema="XXX",
        database="XXX",
        warehouse="COMPUTE_WH",
        credentials=SnowflakeCredentials(
            user=""XXX"",
            password=secret_block.get(),
            account="XXX",
            role="XXX",
        ),
    )
    target_configs = SnowflakeTargetConfigs(
        connector=connector
    )
    dbt_cli_profile = DbtCliProfile(
        name="XXX",
        target="XXX",
        target_configs=target_configs,
    )
    trigger_dbt_cli_command(
        "dbt deps",
        overwrite_profiles=True,
        dbt_cli_profile=dbt_cli_profile
    )
    <http://logger.info|logger.info>("Called dbt deps, calling docs generate.")
    trigger_dbt_cli_command(
        "dbt docs generate",
        overwrite_profiles=True,
        dbt_cli_profile=dbt_cli_profile
    )
    <http://logger.info|logger.info>("Done calling dbt run.")
    
if __name__ == "__main__":
    build_dbt_docs()```

There might be a more efficient solution, but I feel like you could probably import `boto3` and maybe use <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.put_object|put_object()> to cp that up to your desired destination prior to the container exiting?

_May_ also need to setup a new IAM role to have ECS be able to write to s3.

I actually already had the S3 block so I just loaded the block and added:
```s3_block.put_directory('/opt/prefect/target/', 'target/')```
which made it work. Thank you!