One of my flows is struggling to process: agent is on AKS and the CPU usage, through lens, shows a value of 1 for about 6 hours before dropping to negligible amounts (0.002, etc) while still maintaining high levels of RAM usage (30-40gb). At first I assumed it was stalling, but looking further it seems that the flow is only using 1/8 cores at 100%. This explains the continued RAM use.
Any suggestions on how to utilize the full compute power of the aks vm? I'm assuming DaskTaskRunner is the route to go, and I'm currently experimenting with it, but not having much luck with configurations.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.