Thread
#prefect-community
    emre

    emre

    1 year ago
    Hi everyone Although task mapping with DFE feels AMAZING, I have a need to use the old way (BFE) for my ELT pipeline. Is there any way to enforce BFE, either for the entire flow, or more preferably enforce at some mapping levels and get the best of both worlds? My use case is simply financial. My snowflake load operations take 1-2 minutes if run back to back, but DFE distributes the load operations over my 1 hour runtime, thus keeping my snowflake virtual warehouse up and increasing compute cost.
    I guess I can always enforce waiting on all mapped tasks at a level, by inserting a dummy reduce step between mapping levels. But a built in option provided by prefects
    .map()
    would be infinitely more elegant.
    Kyle Moon-Wright

    Kyle Moon-Wright

    1 year ago
    Hey @emre, Rather than an additional task, you can also reduce using the flatten kwarg with your mapping. Though I really like your idea of being able to set DFE or BFE in a kwarg - making it really explicit. We'd love to see an issue with the functionality you'd like to see.
    emre

    emre

    1 year ago
    Glad you like the idea! The docs for
    flatten()
    say that flatten assumes the input is a list of lists, I did not have the chance to try, I am getting an unrelated error that prevents me to start a
    DaskExecutor
    , but this makes me think simply wrapping my task inputs with flatten would result in an error. And I would like to avoid changing my
    return x
    statements to
    return [x]
    😅 .
    Kyle Moon-Wright

    Kyle Moon-Wright

    1 year ago
    Good catch, yep definitely don't add that if you need to change your return statements 😒haking-angry-fist:... Going back to your original question, I think adding a dummy task after each of your maps, as Chris suggests here, is the best way to ensure BFE currently.