https://prefect.io
Join Slack
Hi everyone. We are a small data company based in Europe. We use prefect since months without going...
# ask-community
j
Hi everyone. We are a small data company based in Europe. We use prefect since months without going on cloud. We built all our automation flows on prefect and it's an amazing tool, thanks a lot for your work. We decided to move some flows on the prefect cloud backend and we were surprised to see the total duration time increase by a factor of two compared to our local prefect backend server. The flow on which we ran the tests is one of the longest in our stack (191 tasks, 26 parameters). Keeping the same agent, executor, computer, and parameters. We move from 65s to 135s. Each task takes on average +100% time increase (even the parameters collection). So, we understand that there is some time cost linked to the exchange of state information and metadata but could you, dear fellow community, please help us with the following questions? 1. Is the time gap between runs on the prefect cloud backend and the prefect server backend proportional: The longer the flow, the longer the time gap? 2. Are there any good practices to reduce this time gap? 3. Upgrading to the standard or enterprise offer will that make things faster? Thanks a lot :) Extra information: Our internet connection at the office 127 mbps/s download, 7.8 mbps/s upload.
k
Hi @juumel_team! This is not expected, especially since all of the resources are the same. Just to be sure, are you comparing the tasks one-to-one and all are consistently double the execution time? Do you have mapped tasks here? Have you tried another flow and seen the same behavior?
j
Hi @Kevin Kho, thanks for the answer. Yes we compared the same tasks one-to-one and the execution time double in average (see the capture bellow). There is no mapped tasks in this Flow. And we didn't tried another Flow yet, but I will try one today.
As an example, this task load an ML model, everything is similar on both backend and we can see the execution time is +5s for this task.
k
That’s being loaded from something like S3? It should be the speed of the machine it’s running on so if it’s running on the same machine, I don’t get why the big gap….We’ll talk about this and see how we can debug.
Could you try a simpler flow that loads something smaller and see it’s a constant +5 or a factor increase?
j
No the model is stored locally. Same machine it's currently my laptop. Thanks, I will keep you update, if a gap exist with another flow. Some version info (in case of): 5.10.38-1-MANJARO, Docker 20.10.6, python 3.8.6, prefect 0.13.19
I tried a simpler flow who just send an email. • Prefect server backend: 12s • Prefect cloud backend: 16s
On cloud
On server
In this flow the time gap seems to be linked to the Parameters.
k
Gotcha, Thanks for the additional information
So this additional overhead with requests over the API is more in line with the expectations. I don’t think this indicates that it is a factor increase. Each task run makes a minimum of 1 API request per state change, so at least 3 API requests to cloud.
I would also check if checkpointing settings are the same between your setups just to make sure persisting results are not causing the difference
j
Ok, thanks a lot @Kevin Kho, I will continue a but further the investigation and I will keep you inform if I find something.
👍 1
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by