In one of my flows, I have a task to upload a video to s3 (boto3 s3 resource
upload_file()
call, because big&binary). Since it's synchronous, I can trust the file will be where desired when task succeeds.
Next task in this flow is to transcode this video. I do so by creating a
mediaconvert
boto3
client, and sending a mess of json to its
create_job(**args)
method. This returns me... the job.
Now from what I've read... I should be able to use a prefect built-in
prefect.tasks.aws.client_waiter.AWSClientWait
to wait for said job to finish (which is fine, at this point the workflow is serial/synchronous). Problem is... even when the job reports success (in the console, even!), it takes a while (minutes?!) for the transcoded movie to be present in the target bucket.
I would then... need to enter another wait task until I could find the file in the bucket's list of objects, possibly through
prefect.tasks.aws.s3.S3List? until I could proceed to do further things to this transcoded video?
This conjunction sounds all too common not to have an integrated solution, unless I'm being dense (hah! no news there!) and not spotting an obvious solution. Any guidance?