Hi all, I'm new to Prefect. Can you help me out wi...
# prefect-community
Hi all, I'm new to Prefect. Can you help me out with this: This is the basic flow:
Copy code
def main():
     # list of arguments
     ls = [1,2,3,4]
     # start a thread for calculate method for each arg in ls
     for each in ls:
 def thread(arg):
     t = threading.Thread(target=calculate, args=(arg))
 def calculate(*args):
     # does time consuming calculations and pushes the result in the db
I'm running
method in a python thread, so that multiple threads can run the calculations. Issue is, if
is too long, I crash the server (maybe due to too many threads). I want to do this using prefect-dask, but I'm not sure how to do it, or where to look for examples. Any help will be much appreciated. Thanks.
Hi @Sumant Agnihotri you should definitely take advantage of Dask here to execute your tasks in parallel! To get started have a look as the doc sections on using Executors and dask! https://docs.prefect.io/core/concepts/engine.html#executors https://docs.prefect.io/core/advanced_tutorials/dask-cluster.html https://examples.dask.org/applications/prefect-etl.html
👍 1