Thread
#prefect-server
    j

    Joseph

    1 year ago
    Is there any way to create a one to many mapping between a task A and B, where B depends on A, but B’s mapping dimension is of lower cardinality than A’s so that some instances of B would depend on multiple instances of A?
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    No, you'd need to do a reduction in the middle where all mapped tasks filter into a single task where they could be regrouped into a smaller list before remapping. If your results are small in size this will only be a bottleneck on processing speed (all
    A
    tasks must finish before
    B
    tasks can start), but should still be functional. If the results are large in memory then this may be problematic. You could emulate the behavior you want by manually creating all the A/B tasks you need with the proper graph dependency structure, but that freezes you at registration time into a known number of A/B tasks rather than using mapping which is dynamic.
    (Aside: this is a better question for the
    prefect-community
    channel, since it's not about Prefect Server)
    j

    Joseph

    1 year ago
    Thanks for the answer and the clarification on the channels. I’ll do that in the future.
    When you say that a mapping dimension is “dynamic” do you mean that it can be determined at the time the flow is run?
    (rather than at the time of registration)
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Yeah, the number of tasks created by
    task.map
    is dynamic and determined at runtime.
    j

    Joseph

    1 year ago
    Thanks