Hi all, I'm new to Prefect. Can you help me out with this: This is the basic flow:

def main():
     # list of arguments
     ls = [1,2,3,4]
 
     # start a thread for calculate method for each arg in ls
     for each in ls:
         thread(each)
 
 def thread(arg):
     t = threading.Thread(target=calculate, args=(arg))
     t.start()
 
 def calculate(*args):
     # does time consuming calculations and pushes the result in the db

I'm running

calculate

method in a python thread, so that multiple threads can run the calculations. Issue is, if

ls

is too long, I crash the server (maybe due to too many threads). I want to do this using prefect-dask, but I'm not sure how to do it, or where to look for examples. Any help will be much appreciated. Thanks.

Hi @Sumant Agnihotri you should definitely take advantage of Dask here to execute your tasks in parallel! To get started have a look as the doc sections on using Executors and dask! https://docs.prefect.io/core/concepts/engine.html#executors https://docs.prefect.io/core/advanced_tutorials/dask-cluster.html https://examples.dask.org/applications/prefect-etl.html