hi all, if i have an upstream prefect task submitting a job to aws batch and a downstream task which executes upon the aws batch job completes. is it good to have the downstream task to keep checking whether the aws batch job completes by using boto3 sdk? or is there a more elegant way to achieve this?

Hey @Wai Kiat Tan, you can do something like AWS Batch completion -> Trigger Lambda -> Prefect Create Flow Run Endpoint.

so the lambda in between would trigger another prefect flow instead?

Yes by hitting the GraphQL API

thanks for sharing the thread, what if the upstream prefect task is mapped and i would like to wait for all batch jobs to complete?

Then yeah I think you would need to do that checking with the boto3 sdk unless the Lambdas can handle it before the flow is triggered.