What is the best practice for implementing fan-out-to-fan-in patterns in Prefect 2.0? As a test scenario, I tried to create a flow that 1. Gets all AWS S3 bucket names 2. Spawns a task per bucket and counts the objects 3. Collects all count outputs and calculates the sum (Code is in the first comment) I get some errors (also in the comment) but I have the feeling that I'm using Prefect 2.0 in a way that might be wrong. How should I structure my flow to do it in the most Prefect-y way? I loosely based the idea on this amazing Prefect blog post.

I think the issue is with

bucket_objects = get_buckets + count_bucket_objects(bucket.name)

because

get_buckets

is a Task itself. Shouldn't you be doing something like

bucket_objects += count_bucket_objects(bucket.name)

? Also as an aside, you have a nice task for retrieving all your bucket names but you don't use them in your flow, missing all that exception management you did.

Good catch! I had a typo in this line. After using your suggested code I unfortunately still get an error:

TypeError: unsupported operand type(s) for +=: 'int' and 'PrefectFuture'

Any idea what I am doing wrong?

Ah right. I'm not really used to Prefect 2.0 for now but I believe you can do something like this :

bucket_objects += count_bucket_objects(bucket.name).result()

as referenced here. Alternatively you could create a task to add numbers which would allow you to have the Future's result directly but it feels a bit too verbose for such a simple thing.

You are 100% right. Thanks a lot!

My pleasure !