Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Does anyone know how to run an AWS batch job and wait for completion?

Look at Theo’s thread <https://prefect-community.slack.com/archives/CL09KU1K7/p1636573613348000|here>

He uses <https://docs.prefect.io/api/latest/tasks/aws.html#batchsubmit|BatchSubmit> and <https://docs.prefect.io/api/latest/tasks/aws.html#awsclientwait|AWSClientWait>

i am too, but having issues with the waiter just witing forever

no idea.  doing a very simple flow to figure it out.

this is literally my flow:
```batch_submit = BatchSubmit()

batch_observer = AWSClientWait(client="batch", waier_name="JobCompleted")

with Flow(FLOW_NAME, storage=STORAGE) as flow:
    hello_world_job = batch_submit.run(
        job_name="prefect-tcf-hello",
        job_definition="tcf-sample",
        job_queue="tcf-tokenization",
        # batch_kwargs=boto_args,
    )

    batch_observer()    

    
flow.register(project_name=PROJECT_NAME)```


when I run the flow, it just hangs on the AWSClientWait

You shouldn’t need to call .rrun() for batch_submit. I would try re-registering and re-running without that

so something like this then
```batch_submit = BatchSubmit(
        job_name="prefect-tcf-hello",
        job_definition="tcf-sample",
        job_queue="tcf-tokenization",)

batch_observer = AWSClientWait(client="batch", waiter_name="JobCompleted")

with Flow(FLOW_NAME, storage=STORAGE) as flow:
    hello_world_job = batch_submit()
    batch_observer()    

    
flow.register(project_name=PROJECT_NAME)```


I haven’t used this myself but shouldn’t you point the Wait to the job somehow?

i suspect so.  The thread you pointed me to has some hints that may help.