https://prefect.io logo
Title
m

matta

02/18/2021, 7:30 PM
So, we run our dbt stuff via Prefect. We wanted to be able to see the compiled SQL queries, and found turning them into Flow Artifacts was the easiest way to do it! Shocked by how painless it was. Here's a few functions:
@task(trigger=all_finished)
def get_filepaths(dbt_path: str = "/root/dbt-repo/target") -> t.List[pathlib.PosixPath]:
    return list(pathlib.Path(f"{dbt_path}/compiled").rglob("*.sql"))


def make_query_name(path: pathlib.PosixPath) -> str:
    query_name = str(path).split("/compiled")[1]
    return f"# {query_name}"


def make_sql_markdown(sql: str) -> str:
    return f"
sql\n{sql}\n```" @task(trigger=all_finished) def publish_artifact(filepaths: t.List[pathlib.PosixPath]) -> None: titles_and_queries = [ "\n".join((make_query_name(path), make_sql_markdown(path.read_text()))) for path in filepaths ] all_merged = "\n\n".join(titles_and_queries) create_markdown(all_merged)```
🧐 2
👏 3
💯 5
g

George Coyne

02/18/2021, 7:33 PM
This is very cool
d

Dylan

02/18/2021, 7:35 PM
This is awesome
a

ale

02/18/2021, 7:48 PM
This is REALLY awesome 😍