What is the best practice for implementing fan-out...
# prefect-community
What is the best practice for implementing fan-out-to-fan-in patterns in Prefect 2.0? As a test scenario, I tried to create a flow that 1. Gets all AWS S3 bucket names 2. Spawns a task per bucket and counts the objects 3. Collects all count outputs and calculates the sum (Code is in the first comment) I get some errors (also in the comment) but I have the feeling that I'm using Prefect 2.0 in a way that might be wrong. How should I structure my flow to do it in the most Prefect-y way? I loosely based the idea on this amazing Prefect blog post.
I think the issue is with
bucket_objects = get_buckets + count_bucket_objects(bucket.name)
is a Task itself. Shouldn't you be doing something like
bucket_objects += count_bucket_objects(bucket.name)
? Also as an aside, you have a nice task for retrieving all your bucket names but you don't use them in your flow, missing all that exception management you did.
Good catch! I had a typo in this line. After using your suggested code I unfortunately still get an error:
TypeError: unsupported operand type(s) for +=: 'int' and 'PrefectFuture'
Any idea what I am doing wrong?
Ah right. I'm not really used to Prefect 2.0 for now but I believe you can do something like this :
bucket_objects += count_bucket_objects(bucket.name).result()
as referenced here. Alternatively you could create a task to add numbers which would allow you to have the Future's result directly but it feels a bit too verbose for such a simple thing.
Basically what I understood so far is that calling tasks yields
objects. The actual result values behind these can be accessed either by passing the Future as parameter to another Task or by calling
You are 100% right. Thanks a lot!
My pleasure !