Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi to all,
I'm trying to use Prefect Jobs as an orchestrator for AWS GLUE Jobs. Does anyone try to implement start of aws glue jobs &amp; monitoring the execution?

Hi <@U017JPD6Y8N>! I haven’t tried it myself but it should work with boto3. For monitoring you may be interested in looking at the extra loggers: <https://docs.prefect.io/core/concepts/logging.html#extra-loggers>

<@U017JPD6Y8N> I’ll be doing the same soon. Please let us know how it goes :smile:

Hi <@U01TXEHS3R9>, I'm planning to refactor Prefect AWS Tasks <https://github.com/PrefectHQ/prefect/tree/master/src/prefect/tasks/aws>

For template will take lambda_function.py script

Also plan to add some tasks for AWS Athena through Boto3 too

Sounds good! <@U017JPD6Y8N> Looking forward to it