Hi all, I was wondering if someone could help me identify why only part of my flow executes in parallel. As shown in this flow diagram, 5/8 tasks have been mapped, while 3 are still pending. I am sure that I started enough Dask worker nodes to process the compute, but these jobs are stuck pending. Thanks for the advice !
m
Marwan Sarieddine
12/13/2020, 1:38 AM
Hi Vincent, perhaps your dask scheduler is resource constrained ?
v
Vincent
12/13/2020, 10:52 PM
I think I may have figured out why this occurred. Some of the nodes used were blocked from communicating with graphql service. This likely caused the pods to exit without further communication to the scheduler.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.