which is mapped to 180 files. It takes 30 seconds when I run this flow in local without agent and it takes 2 mins when i run it using local agent by UI. What can i do for optimizing this point. Thanks a lot.
k
Kevin Kho
09/27/2021, 4:26 PM
Hey @Qin XIA, there will be some latency with each task because each task has to hit the Prefect API for each state change. That means for each task, there will be at least 3 API calls. It would probably be faster if you batch your files together to limit API calls, but then you lose the observability.
The API calls are a fixed overhead. For longer tasks, they will be less noticeable.
q
Qin XIA
09/27/2021, 4:30 PM
@Kevin Kho i will try to review my conception. THX
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.