Hello!
I have a question regarding how resources are being used in Prefect. I am using the DaskExecutor with LocalCluster(2 workers, and 4 threads per worker) and am testing processing 800 files which I am splitting evenly into 8 batches.
Originally I had the prefect task running each of the batches passed to them and processing those in a loop, but when I change this to explicitly run threading within the tasks (still mapping to them) using the existing client like how Kevin suggested in this
POST, I am seeing significant time savings (25 - 50% faster).
My question is are there drawbacks to performing batch processing this way/is this the right way to do this or were we not utilizing Prefect in the correct way previously?