https://prefect.io logo
Title
n

Nico Aiello

02/26/2020, 9:17 PM
. --------> item a1 ------------> batch a --------> item a2 batcher --------> item a3 ------------> batch b --------> item b1 --------> item b2 Basically, I'd like prefect to be aware that once the batching happens, we now have two independent, parallelizable flows happening - so if batch a finishes, item a1 can begin even if batch b is still in progress.
j

Jeremiah

02/26/2020, 9:21 PM
Hey @Nico Aiello - first of all, welcome to the community! What your describing is a feature we call “depth-first execution,” and should function exactly as you describe. We’ve prioritized it for an upcoming release: https://github.com/PrefectHQ/prefect/issues/2041
n

Nico Aiello

02/26/2020, 9:29 PM
Thanks for replying! I saw a few days ago DFE being mentioned and it definitely sounds like that's what I'm after. One thing I'm still having trouble coding up in prefect is this concept of one tasking leading to multiple identical tasks which each in turn lead to multiple tasks. I thought it would be something like:
batches = get_batches()
items = run_batch.map(batches)
run_item.map(items)
but this third line does not do what I want - it runs on the entire list of items (so in my example [a1, a2, a3] and [b1, b2]) instead of running individually on each of a1, a2, a3, b1, b2. Does that make sense?
j

Jeremiah

02/26/2020, 9:36 PM
It does make sense. We actually don’t have a one-liner for handling your exact branching case right now (some people call it a
flat_map
, and we’ve been talking about implementing one). Normally, I would advise you to add a task to your flow whose job was to intercept the output of your first map step, flatten it, and then have your second task map over that flattened output. However, that might not play nicely with our depth-first semantics, since the “flatten” step will block until all upstream results are complete. With the current version of Prefect my advice is what I just wrote, and I want to think more about your branching case as we write tests for depth-first execution.
n

Nico Aiello

02/26/2020, 9:38 PM
Thanks! This was really helpful.
j

Jeremiah

02/26/2020, 9:39 PM
Absolutely! The mapping feature is under HEAVY work right now so you should see these improvements start to roll out in the near future.
n

Nico Aiello

02/26/2020, 9:41 PM
Amazing. I'll code up this flatten idea and then come back after incorporating the DFE stuff after it's released. Thanks again!
j

Jeremiah

02/26/2020, 9:54 PM
Great — I’ve opened a descriptive issue here https://github.com/PrefectHQ/prefect/issues/2091
h

Henrique Guarnieri

02/26/2020, 11:49 PM
This is amazing, I have just joined the slack channel to have some help on the same thing and it worked like a charm. Thanks, @Jeremiah!
👍 1
n

Nico Aiello

03/10/2020, 2:41 PM
Very excited for the 'flat_map' DFE compatible operator linked above. Just out of curiosity, do you have some rough idea of when we can expect to see this released?
j

Jeremiah

03/10/2020, 8:10 PM
Hi Nico, DFE itself is expected this month and the flatmap should follow
n

Nico Aiello

03/10/2020, 8:17 PM
Awesome! Thanks!