Hey all, we observe some odd behavior were prefect spawns lots of
prefect.engine
tasks in the EKS kubernetes pods when we attempt to run a deployment with higher parallel number of runs.
We observe that flows take much longer than when running them e.g. locally.
So, the questions are:
• why does prefect spawn these
prefect.engine
tasks?
• Do they relate to the threadpool executor?
• It seems like they are eating up CPU time, so that there is no CPU left to do the actual processing. Could this be true?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
