I have started using Prefect for the first time, and now I am in a dilemma whether to continue using it or not.
I am developing a high performance REST API, and it has ETL functionality, so I thought Prefect would be best to orchestrate the whole workflow.
Anyway, the problem is that with the same code, the response time of the API varies drastically based on I am using prefect or not. (I understand that the first run of the workflow will be very slow, but I have a problem with the subsequent invocations being very slow). I am attaching images of both scenarios here.
My question is whether I am doing something wrong, or is it something I will have to live with if I decide to use Prefect?
@Vikas do you use prefect server and agent to run your flows or you just run them with python?
09/21/2022, 9:38 AM
@max just with Python, inside scripts
09/21/2022, 10:12 AM
probably you will get better performance with using prefect server and agents
prefect server - is the main thing that work with your flows. you should run it first, then deploy flows in there and create a work-queue and agent to run your flows.
now you don't get the best performance because running flows in scripts is needed just for tests (if I understand prefect correctly 😄)
09/21/2022, 10:14 AM
I will try it with the correct approach then