Does anyone know how to run an AWS batch job and wait for completion?

# ask-community

Keith Veleba

01/31/2022, 5:53 PM

Does anyone know how to run an AWS batch job and wait for completion?

Kevin Kho

01/31/2022, 5:53 PM

Look at Theo’s thread here

Keith Veleba

01/31/2022, 5:56 PM

thx

Kevin Kho

01/31/2022, 5:57 PM

He uses BatchSubmit and AWSClientWait

Keith Veleba

01/31/2022, 5:57 PM

i am too, but having issues with the waiter just witing forever

Kevin Kho

01/31/2022, 5:58 PM

Are you using mapped?

Keith Veleba

01/31/2022, 5:58 PM

no idea. doing a very simple flow to figure it out.

Keith Veleba

01/31/2022, 6:01 PM

this is literally my flow:

Copy code

batch_submit = BatchSubmit()

batch_observer = AWSClientWait(client="batch", waier_name="JobCompleted")

with Flow(FLOW_NAME, storage=STORAGE) as flow:
    hello_world_job = batch_submit.run(
        job_name="prefect-tcf-hello",
        job_definition="tcf-sample",
        job_queue="tcf-tokenization",
        # batch_kwargs=boto_args,
    )

    batch_observer()    

    
flow.register(project_name=PROJECT_NAME)

Keith Veleba

01/31/2022, 6:02 PM

when I run the flow, it just hangs on the AWSClientWait

Keith Veleba

01/31/2022, 6:02 PM

there are very few examples

Kevin Kho

01/31/2022, 6:04 PM

You shouldn’t need to call .rrun() for batch_submit. I would try re-registering and re-running without that

Keith Veleba

01/31/2022, 6:09 PM

so something like this then

Copy code

batch_submit = BatchSubmit(
        job_name="prefect-tcf-hello",
        job_definition="tcf-sample",
        job_queue="tcf-tokenization",)

batch_observer = AWSClientWait(client="batch", waiter_name="JobCompleted")

with Flow(FLOW_NAME, storage=STORAGE) as flow:
    hello_world_job = batch_submit()
    batch_observer()    

    
flow.register(project_name=PROJECT_NAME)

Kevin Kho

01/31/2022, 6:12 PM

I haven’t used this myself but shouldn’t you point the Wait to the job somehow?

Keith Veleba

01/31/2022, 6:13 PM

i suspect so. The thread you pointed me to has some hints that may help.

21 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.