Hi all, I have a question. I am trying to connect to the latest version of prefect 3 in my project. At present, my usage is still very simple. The python service creates process nodes through annotations such as @flow and @task, and uses artifact to report data to the prefect API server. But I found that it has a great impact on performance. Is there any good way to improve performance?
Before joining Prefect
cost: 16 ms
After joining Prefect
cost: 1083 ms
j
Janet Carson
11/11/2024, 5:35 PM
Is this a toy test case? If the job only takes 16ms, I'm not sure why you would want to add the overhead of orchestration in the first place? There will be overhead, but the idea is that the observability and other features added by prefect will be worth it for jobs that take a long time to run. (1 second of overhead is tiny if your total job takes 5 hours, say)
d
Darcy
11/12/2024, 5:27 AM
Thanks for your reply. The performance data is obtained through a toy test case. It seems that prefect is mainly suitable for offline long task execution.🙏
Darcy
11/12/2024, 5:31 AM
The reason for this is that we are using a very lengthy YAML for process orchestration. When we saw that Prefect can define processes with code and is easy to integrate, we started to try it.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.