Hi everyone Although task mapping with DFE feels AMAZING, I have a need to use the old way (BFE) for my ELT pipeline. Is there any way to enforce BFE, either for the entire flow, or more preferably enforce at some mapping levels and get the best of both worlds? My use case is simply financial. My snowflake load operations take 1-2 minutes if run back to back, but DFE distributes the load operations over my 1 hour runtime, thus keeping my snowflake virtual warehouse up and increasing compute cost.

Hey @emre, Rather than an additional task, you can also reduce using the flatten kwarg with your mapping. Though I really like your idea of being able to set DFE or BFE in a kwarg - making it really explicit. We'd love to see an issue with the functionality you'd like to see.

Glad you like the idea! The docs for

flatten()

say that flatten assumes the input is a list of lists, I did not have the chance to try, I am getting an unrelated error that prevents me to start a

DaskExecutor

, but this makes me think simply wrapping my task inputs with flatten would result in an error. And I would like to avoid changing my

return x

statements to

return [x]

😅 .

Good catch, yep definitely don't add that if you need to change your return statements shaking angry fist... Going back to your original question, I think adding a dummy task after each of your maps, as Chris suggests here, is the best way to ensure BFE currently.