https://prefect.io logo
e

emre

10/19/2020, 1:45 PM
Hi everyone Although task mapping with DFE feels AMAZING, I have a need to use the old way (BFE) for my ELT pipeline. Is there any way to enforce BFE, either for the entire flow, or more preferably enforce at some mapping levels and get the best of both worlds? My use case is simply financial. My snowflake load operations take 1-2 minutes if run back to back, but DFE distributes the load operations over my 1 hour runtime, thus keeping my snowflake virtual warehouse up and increasing compute cost.
I guess I can always enforce waiting on all mapped tasks at a level, by inserting a dummy reduce step between mapping levels. But a built in option provided by prefects
.map()
would be infinitely more elegant.
k

Kyle Moon-Wright

10/19/2020, 3:33 PM
Hey @emre, Rather than an additional task, you can also reduce using the flatten kwarg with your mapping. Though I really like your idea of being able to set DFE or BFE in a kwarg - making it really explicit. We'd love to see an issue with the functionality you'd like to see.
e

emre

10/19/2020, 7:44 PM
Glad you like the idea! The docs for
flatten()
say that flatten assumes the input is a list of lists, I did not have the chance to try, I am getting an unrelated error that prevents me to start a
DaskExecutor
, but this makes me think simply wrapping my task inputs with flatten would result in an error. And I would like to avoid changing my
return x
statements to
return [x]
😅 .
k

Kyle Moon-Wright

10/19/2020, 7:55 PM
Good catch, yep definitely don't add that if you need to change your return statements shaking angry fist... Going back to your original question, I think adding a dummy task after each of your maps, as Chris suggests here, is the best way to ensure BFE currently.
2 Views