Is there any way to create a one to many mapping between a task A and B, where B depends on A, but B’s mapping dimension is of lower cardinality than A’s so that some instances of B would depend on multiple instances of A?
1 year ago
No, you'd need to do a reduction in the middle where all mapped tasks filter into a single task where they could be regrouped into a smaller list before remapping. If your results are small in size this will only be a bottleneck on processing speed (all
tasks must finish before
tasks can start), but should still be functional. If the results are large in memory then this may be problematic. You could emulate the behavior you want by manually creating all the A/B tasks you need with the proper graph dependency structure, but that freezes you at registration time into a known number of A/B tasks rather than using mapping which is dynamic.
(Aside: this is a better question for the
channel, since it's not about Prefect Server)
1 year ago
Thanks for the answer and the clarification on the channels. I’ll do that in the future.