Hey All,
Does anyone know a way to speed up prefect flows start? They seem to stay in scheduled or pending for a significant among of time. My flows are served using the .serve() method. I would think it should be able to begin code execution immediately when the deployment is invoked/triggered?
I’m considering moving a lot of logic into an API +Redis + celery system because of how slow it is. I guess it’s likely I’m not using prefect for an ideal use case.
c
Chris White
11/29/2023, 1:34 PM
Hi Will - thanks for raising this; it should begin immediately so it sounds like there is something up that we should investigate. I've personally seen this before whenever the subprocess takes a long time to get started.
Could you share some details about your setup? Things like version, how many flows you're running, do you use
from_source
, and anything else you think could be useful for me to know?
w
will hitchcock
12/27/2023, 2:56 AM
@Chris White hey, sorry for the slow response, I’m not sure what details are relevant. We have maybe 3 flows. We are running one at a time although multiple are in one script. We use version 2.14.3. This is in kubernetes and flows executed as sub processes. We don’t specify a from source. It can be up to a 10 second lag from invocation until work actually begins.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.