@Marvin what do the

prefect.engine

processes refer to? Seems like there is one processes per thread? Could it be that it consumes all the available CPU time in AWS resources?

The

prefect.engine

module is responsible for executing flow and task runs in Prefect. It manages the orchestration of tasks, logging, and flow control. The engine operates by running flows either synchronously or asynchronously, as indicated by the flow's configuration. Here's a link to the engine code for more details. Regarding CPU usage on AWS, it's important to note that Prefect workers typically submit work through API calls to AWS rather than executing flow code directly. This means the CPU and memory allocations for workers are generally minimal. If you're experiencing high CPU usage, it might be due to the specific tasks being executed or how the flows are structured rather than the engine itself. You can refer to the AWS integration guide for more insights on managing resources effectively in AWS. If you're facing performance issues, consider reviewing your flow's configuration and the tasks it executes to ensure they're optimized for the resources available.