Donny Flynn
02/09/2023, 5:45 PMdbt docs generate
as the Flow Run is outputting that
Catalog written to /opt/prefect/target/catalog.json
I checked S3 and it's definitely not there, and I'm guessing that the /opt
is a directory that's tied to the ECS Task from the container? Is there a way that the dbt cli command can output the docs file (specifically index.html into the S3 bucket so we can host our dbt docs as a static site)
Flow code is in the ๐งต. I really appreciate any help or pointers ๐from prefect import task, flow
from prefect import get_run_logger
from prefect import flow
from prefect_dbt.cli.commands import trigger_dbt_cli_command
from prefect_snowflake.credentials import SnowflakeCredentials
from prefect_snowflake.database import SnowflakeConnector
from prefect_dbt.cli.commands import trigger_dbt_cli_command
from prefect_dbt.cli.configs import SnowflakeTargetConfigs
from prefect_dbt.cli import DbtCliProfile
from prefect.blocks.system import Secret
@flow
def build_dbt_docs():
logger = get_run_logger()
<http://logger.info|logger.info>("Calling dbt deps")
# Access the stored secret
secret_block = Secret.load("XXX")
connector = SnowflakeConnector(
schema="XXX",
database="XXX",
warehouse="COMPUTE_WH",
credentials=SnowflakeCredentials(
user=""XXX"",
password=secret_block.get(),
account="XXX",
role="XXX",
),
)
target_configs = SnowflakeTargetConfigs(
connector=connector
)
dbt_cli_profile = DbtCliProfile(
name="XXX",
target="XXX",
target_configs=target_configs,
)
trigger_dbt_cli_command(
"dbt deps",
overwrite_profiles=True,
dbt_cli_profile=dbt_cli_profile
)
<http://logger.info|logger.info>("Called dbt deps, calling docs generate.")
trigger_dbt_cli_command(
"dbt docs generate",
overwrite_profiles=True,
dbt_cli_profile=dbt_cli_profile
)
<http://logger.info|logger.info>("Done calling dbt run.")
if __name__ == "__main__":
build_dbt_docs()
Aaron Gonzalez
02/09/2023, 5:55 PMboto3
and maybe use put_object() to cp that up to your desired destination prior to the container exiting?Donny Flynn
02/09/2023, 7:17 PMs3_block.put_directory('/opt/prefect/target/', 'target/')
which made it work. Thank you!