Thread
#prefect-community
    j

    Jacques

    2 years ago
    Sorry, got one more question ๐Ÿ™‚ - looking at the ETL examples where you are doing something like extracting a list of values, mapping them to a transform (map reduce type operation) and then finally using the reduced transform result load that into e.g. a database. Is there a way to have this fan-out instead, in other words not have the reduce function for the map and end with one task. Not sure if that makes sense, so would something like 1 extract task produces 10 results, and that kicks off 10 parallel transforms, each producing one output that is then passed to 10 parallel load tasks be possible?
    nicholas

    nicholas

    2 years ago
    Hi @Jacques definitely! The reduce step on
    .map()
    is completely optional; if I understand you correctly, you could map again over the results of the transform
    .map()
    to make 10 separate load tasks.
    j

    Jacques

    2 years ago
    Ok I think I missed an important piece here. If some mapped transforms take 1s and others 5 mins and I do the map over the result of the map then would it need to wait for the last transform to complete before starting the loads?
    a

    Alex Cano

    2 years ago
    @Jacques Currently, yes. However, theyโ€™re working on implementing depth first execution (dfe), which would allow for the 1 second task to no longer depend on the 5 minute task to complete.
    nicholas

    nicholas

    2 years ago
    @Jacques as @Alex Cano says, yes, that's something we're actively working on (there's an issue here: https://github.com/PrefectHQ/prefect/issues/2041)
    j

    Jacques

    2 years ago
    Epic, thanks, I'll watch that issue!