k

    KIRYL BUCHA

    1 year ago
    Hi to all, I'm trying to use Prefect Jobs as an orchestrator for AWS GLUE Jobs. Does anyone try to implement start of aws glue jobs & monitoring the execution?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @KIRYL BUCHA! I haven’t tried it myself but it should work with boto3. For monitoring you may be interested in looking at the extra loggers: https://docs.prefect.io/core/concepts/logging.html#extra-loggers
    David Vasandani

    David Vasandani

    1 year ago
    @KIRYL BUCHA I’ll be doing the same soon. Please let us know how it goes 😄
    k

    KIRYL BUCHA

    1 year ago
    Hi @David Vasandani, I'm planning to refactor Prefect AWS Tasks https://github.com/PrefectHQ/prefect/tree/master/src/prefect/tasks/aws
    For template will take lambda_function.py script
    Also plan to add some tasks for AWS Athena through Boto3 too
    Kevin Kho

    Kevin Kho

    1 year ago
    Sounds good! @KIRYL BUCHA Looking forward to it