w

    Wai Kiat Tan

    1 year ago
    hi all, if i have an upstream prefect task submitting a job to aws batch and a downstream task which executes upon the aws batch job completes. is it good to have the downstream task to keep checking whether the aws batch job completes by using boto3 sdk? or is there a more elegant way to achieve this?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Wai Kiat Tan, you can do something like AWS Batch completion -> Trigger Lambda -> Prefect Create Flow Run Endpoint.
    w

    Wai Kiat Tan

    1 year ago
    so the lambda in between would trigger another prefect flow instead?
    Kevin Kho

    Kevin Kho

    1 year ago
    Yes by hitting the GraphQL API
    See this
    w

    Wai Kiat Tan

    1 year ago
    thanks for sharing the thread, what if the upstream prefect task is mapped and i would like to wait for all batch jobs to complete?
    Kevin Kho

    Kevin Kho

    1 year ago
    Then yeah I think you would need to do that checking with the boto3 sdk unless the Lambdas can handle it before the flow is triggered.