Hello there! If there a way to perform a mapping with a specific batch size ? for example lets say I have two tasks

f

and `g`:

ys = f.map(xs)
zs = g.map(ys)

Is there a generic way to give

g

batches of y values, for example 10, rather than one at a time ? (And obviously, without having to wait for all the ys to be computed.) Or is this something that is specific to the executor used ?

Hi @Didier Marin, we're working on some features that will make this possible, but right now no there is not. Specific executors like Dask might allow some degree of configuration here This discussion might be helpful https://github.com/PrefectHQ/prefect/issues/2459 If the

ys

values are small enough to fit in memory, it may be possible to implement this batching yourself

Very clear, thank you Zach !