Prefect runs recurring tasks, and during the first few hours, there are no issues. After approximately 24 hours, the server starts freezing, and CPU usage reaches its maximum. We added timeouts on all flows and tasks.
Since the same tasks are executed in parallel at the beginning and the end, I don’t understand why CPU and resource usage start to overload (as if a task is not finishing).
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.