Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

yes. if you can write it in python, you can coordinate/orchestrate it with Prefect

my favorite way of doing it is with <https://aws-sdk-pandas.readthedocs.io/en/stable/tutorials/005%20-%20Glue%20Catalog.html>

```# you could do

from prefect import flow

@flow
def write_to_s3_glue():
    wr.s3.to_parquet(
    df=df,
    path=f"s3://{bucket}/products/",
    dataset=True,
    database="awswrangler_test",
    table="products",
    mode="overwrite",
    description=desc,
    parameters=param,
    columns_comments=comments
)```

I was looking at Airbyte earlier but seems the functionality is quite limited to EL. Additionally I would need to run a server to host it(cost consideration). Therefore I'm revisiting Glue and glad to see that prefect can coordinate it's usage.