https://prefect.io logo
Powered by
k

Keith Veleba

01/31/2022, 5:53 PM
Does anyone know how to run an AWS batch job and wait for completion?
k

Kevin Kho

01/31/2022, 5:53 PM
Look at Theo’s thread here
k

Keith Veleba

01/31/2022, 5:56 PM
thx
k

Kevin Kho

01/31/2022, 5:57 PM
k

Keith Veleba

01/31/2022, 5:57 PM
i am too, but having issues with the waiter just witing forever
k

Kevin Kho

01/31/2022, 5:58 PM
Are you using mapped?
k

Keith Veleba

01/31/2022, 5:58 PM
no idea. doing a very simple flow to figure it out.
this is literally my flow:
Copy code
batch_submit = BatchSubmit()

batch_observer = AWSClientWait(client="batch", waier_name="JobCompleted")

with Flow(FLOW_NAME, storage=STORAGE) as flow:
    hello_world_job = batch_submit.run(
        job_name="prefect-tcf-hello",
        job_definition="tcf-sample",
        job_queue="tcf-tokenization",
        # batch_kwargs=boto_args,
    )

    batch_observer()    

    
flow.register(project_name=PROJECT_NAME)
when I run the flow, it just hangs on the AWSClientWait
there are very few examples
k

Kevin Kho

01/31/2022, 6:04 PM
You shouldn’t need to call .rrun() for batch_submit. I would try re-registering and re-running without that
k

Keith Veleba

01/31/2022, 6:09 PM
so something like this then
Copy code
batch_submit = BatchSubmit(
        job_name="prefect-tcf-hello",
        job_definition="tcf-sample",
        job_queue="tcf-tokenization",)

batch_observer = AWSClientWait(client="batch", waiter_name="JobCompleted")

with Flow(FLOW_NAME, storage=STORAGE) as flow:
    hello_world_job = batch_submit()
    batch_observer()    

    
flow.register(project_name=PROJECT_NAME)
k

Kevin Kho

01/31/2022, 6:12 PM
I haven’t used this myself but shouldn’t you point the Wait to the job somehow?
k

Keith Veleba

01/31/2022, 6:13 PM
i suspect so. The thread you pointed me to has some hints that may help.
16 Views
Powered by