Hi everyone,
I am using dask local executor to run...
# ask-communitya
Ankit
02/03/2023, 6:51 AMHi everyone,
I am using dask local executor to run my flow with 8 threads
Copy code
task_runner=DaskTaskRunner(cluster_class=distributed.LocalCluster, cluster_kwargs={"n_workers": 8})Ankit
02/03/2023, 12:33 PMI tried using the default Concurrent task runner as well, task gets completed after way long that it actually takes to run.
I had 2 queries both of which using SQLAlchemy blocks for db connections, after the first query finishes, next step is to run the second query. But there was a difference of 1 minute between first query completion and second query starting.
Is there anything I am missing out in some configuration?
And not just this, queries are taking longer to run that running the task as a function in jupyter
Ankit
02/03/2023, 1:51 PMI am using KuberntestJob and this issue only happens there, I tried running the flow from jupyter with same env and task completes faster but there is still a 30 seconds lag between queries
Ankit
02/03/2023, 2:42 PMI switched back to dask and checked logs in k8s pod and got this. Is this something related to prefect?
Copy code
14:41:17.588 | INFO | distributed.core - Event loop was unresponsive in Worker for 4.40s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.c
Christopher Boyd
02/03/2023, 3:37 PMConcurrent is not true concurrency, it’s async unless it becomes blocked, so that checks out
Christopher Boyd
02/03/2023, 3:37 PMFor Dask, it’s in distributed.core - which is the dask module. It depends on what kind / size of data you are passing around
a
Ankit
02/04/2023, 8:25 AMI am passing pandas dataframes, both large (over 10M rows x 5 columns) and small (0.1M x 2) takes long time to free up the dask executors