https://prefect.io logo
Title
z

Zack

12/02/2022, 12:52 AM
Can Prefect run AWS glue?
2
a

Anna Geller

12/02/2022, 12:54 AM
yes. if you can write it in python, you can coordinate/orchestrate it with Prefect
# you could do

from prefect import flow

@flow
def write_to_s3_glue():
    wr.s3.to_parquet(
    df=df,
    path=f"s3://{bucket}/products/",
    dataset=True,
    database="awswrangler_test",
    table="products",
    mode="overwrite",
    description=desc,
    parameters=param,
    columns_comments=comments
)
🙌 1
z

Zack

12/02/2022, 1:05 AM
Oh, that's very nice!
I was looking at Airbyte earlier but seems the functionality is quite limited to EL. Additionally I would need to run a server to host it(cost consideration). Therefore I'm revisiting Glue and glad to see that prefect can coordinate it's usage.
a

Anna Geller

12/02/2022, 1:38 AM
💯