. --------> item a1 ------------> batch a --------> item a2 batcher --------> item a3 ------------> batch b --------> item b1 --------> item b2 Basically, I'd like prefect to be aware that once the batching happens, we now have two independent, parallelizable flows happening - so if batch a finishes, item a1 can begin even if batch b is still in progress.

Hey @Nico Aiello - first of all, welcome to the community! What your describing is a feature we call “depth-first execution,” and should function exactly as you describe. We’ve prioritized it for an upcoming release: https://github.com/PrefectHQ/prefect/issues/2041

Thanks for replying! I saw a few days ago DFE being mentioned and it definitely sounds like that's what I'm after. One thing I'm still having trouble coding up in prefect is this concept of one tasking leading to multiple identical tasks which each in turn lead to multiple tasks. I thought it would be something like:

batches = get_batches()
items = run_batch.map(batches)
run_item.map(items)

but this third line does not do what I want - it runs on the entire list of items (so in my example [a1, a2, a3] and [b1, b2]) instead of running individually on each of a1, a2, a3, b1, b2. Does that make sense?

It does make sense. We actually don’t have a one-liner for handling your exact branching case right now (some people call it a

flat_map

, and we’ve been talking about implementing one). Normally, I would advise you to add a task to your flow whose job was to intercept the output of your first map step, flatten it, and then have your second task map over that flattened output. However, that might not play nicely with our depth-first semantics, since the “flatten” step will block until all upstream results are complete. With the current version of Prefect my advice is what I just wrote, and I want to think more about your branching case as we write tests for depth-first execution.

Thanks! This was really helpful.

Absolutely! The mapping feature is under HEAVY work right now so you should see these improvements start to roll out in the near future.

Amazing. I'll code up this flatten idea and then come back after incorporating the DFE stuff after it's released. Thanks again!

Great — I’ve opened a descriptive issue here https://github.com/PrefectHQ/prefect/issues/2091

This is amazing, I have just joined the slack channel to have some help on the same thing and it worked like a charm. Thanks, @Jeremiah!

Very excited for the 'flat_map' DFE compatible operator linked above. Just out of curiosity, do you have some rough idea of when we can expect to see this released?

Hi Nico, DFE itself is expected this month and the flatmap should follow

Awesome! Thanks!