Hi to all,
I'm trying to use Prefect Jobs as an or...
# ask-communityk
KIRYL BUCHA
04/10/2021, 11:51 AMHi to all,
I'm trying to use Prefect Jobs as an orchestrator for AWS GLUE Jobs. Does anyone try to implement start of aws glue jobs & monitoring the execution?
k
Kevin Kho
04/10/2021, 2:51 PM
Hi @KIRYL BUCHA! I haven’t tried it myself but it should work with boto3. For monitoring you may be interested in looking at the extra loggers: https://docs.prefect.io/core/concepts/logging.html#extra-loggers
d
David Vasandani
04/10/2021, 10:18 PM@KIRYL BUCHA I’ll be doing the same soon. Please let us know how it goes 😄
😆 1
👍 1
k
KIRYL BUCHA
04/13/2021, 6:56 AMHi @David Vasandani, I'm planning to refactor Prefect AWS Tasks https://github.com/PrefectHQ/prefect/tree/master/src/prefect/tasks/aws
KIRYL BUCHA
04/13/2021, 6:57 AMFor template will take lambda_function.py script
KIRYL BUCHA
04/13/2021, 6:57 AMAlso plan to add some tasks for AWS Athena through Boto3 too
k
Kevin Kho
04/13/2021, 1:44 PM
Sounds good! @KIRYL BUCHA Looking forward to it
3 Views