Brett Naul
07/07/2020, 9:19 PMtask.map(sequence, x=unmapped(some_large_result))
a bad idea? we're seeing UserWarning: Large object of size 874.55 MB detected in task graph
and very very slow creation of mapped tasks; I would have expected that these would be re-using the dask futures already present on the cluster, not rehydrating the outputs inside the FlowRunner
loop, but I guess that's not how it works. is this expected behavior and if so is there any known workaround..?Jim Crist-Harif
07/07/2020, 9:20 PMsome_large_result
is not the result of a task (e.g. if it's a constant).
If it is the result of a task, we should be able to make the efficient, but the flow runner might be doing something dumb.Brett Naul
07/07/2020, 9:24 PM.submit
ting the actual result bytes for every taskJim Crist-Harif
07/07/2020, 9:26 PMBrett Naul
07/07/2020, 9:27 PMJim Crist-Harif
07/07/2020, 9:52 PM