hello again, one more question, is there a way to ...
# prefect-communityc
Chris Eberly
10/16/2019, 5:12 PMhello again, one more question, is there a way to “yield” a batch of results to start feeding to a mapping process? the example would be, in your guys’ x-files tutorial, if we got say 10 episode urls at a time, can we start the scraping process asynchronously and then collect the results at the end? apologies if i’m missing something obvious, or if this is not a good use case
Chris Eberly
10/16/2019, 5:14 PMthe context of this is reading ~thousands of urls in batches, and there is no need to wait until that finishes to start processing them
c
Chris White
10/16/2019, 5:49 PM
Hey Chris! this is our white whale right now; we call it “depth first mapping” and our current bottleneck is described here: https://stories.dask.org/en/latest/prefect-workflows.html#pain-points-when-using-dask
Chris White
10/16/2019, 5:49 PM
we definitely revisit this regularly and we have a strong goal of supporting this
c
Chris Eberly
10/16/2019, 5:50 PMthat’s awesome, let me know if i can help
c
Chris White
10/16/2019, 6:00 PM
will do!
Chris White
10/16/2019, 6:39 PM
feel free to dig into the task mapping code within the
task_runner.py👍 1