Slava Shor
03/10/2022, 1:24 PMasync def...
?Sylvain Hazard
03/10/2022, 1:27 PMSlava Shor
03/10/2022, 1:43 PMAnna Geller
03/10/2022, 2:16 PMasync
strictly *inside your task*s. Split your lat-lon pairs into, say, batches of size 100. Then every task can download its own batch of size 100 asynchronously.
• prefects unit of work is a task. And dask won't run these tasks asynchronously, so async isn't a viable option to concurrently execute a lot of prefect tasks. But inside the task async can be utilized, no issues.
• note that task Results are held in memory after flows complete and having a high number of futures in Dask can be indeed too memory-intensive so batching would help with thatSarah Floris
03/10/2022, 3:17 PMAnna Geller
03/10/2022, 3:25 PMSarah Floris
03/10/2022, 3:26 PMZanie
03/10/2022, 3:55 PMSlava Shor
03/10/2022, 5:45 PM#
instead of the ?
at the end, so front-end does not render a thing