Luis Muniz
06/26/2020, 7:17 PMmap()
? is using map()
to handle a list of thousands (or millions?) of elements one by one reasonable? Should I micro-batch it into chunks? Does prefect start to choke if you have many many tasks in a flow? What about the Dashboard when you examine such a flow after it has run?nicholas
06/26/2020, 7:19 PM.map
is great for handling lists in the thousands but I think as you go beyond that (or even at that) batching becomes really valuable. In particular as you scale up, you'll want to take a look at parallelization and depth-first execution on Dask; you'll see really improved performance with the latter in particular.Luis Muniz
06/26/2020, 7:21 PMnicholas
06/26/2020, 7:22 PMLuis Muniz
06/26/2020, 7:22 PMnicholas
06/26/2020, 7:22 PMLuis Muniz
06/26/2020, 7:24 PM