Thread
#prefect-server
    Alexis Lucido

    Alexis Lucido

    4 months ago
    Hi, I have been asking for help some time ago, regarding a issue of tasks and flows getting longer and longer to complete until the whole application freeze (my processors being filled with many processes). I have been told to dockerize the app through the guide provided in this link, which I have done, also because it made our deployment process cleaner and more robust. That was quite some work but it has been making our job a lot easier. Unfortunately, we can still observe the same pattern today, though less extreme. I have joined a screenshot of a flow. It almost always fails because it maps a task to fetch data from different sources, and one of them is down at the moment. The flow takes longer to complete not because of the volume of data to fetch. Indeed, it gathers data on a daily basis: every half-hour, it fetches all the data, accumulating throughout the day, for the current date, and it takes longer today at 1am (without much data then) than yesterday at 11pm. When i redeploy the application, the flows gets back to the normal in terms of speed. I've done that this morning, which is why the bars are shorter at the end of the screenshot. I am kinda lost, as we risk, without close monitoring, a failure of our whole environment. The issue makes me think of the Airflow's instable scheduler one. As a matter of fact, the fix I've thought of is to redeploy the application nightly, but I would by far better understand the root cause of the issue. If you have any idea, I'd be delighted to make some further research with you, and maybe we could help Prefect become an even better product. Alexis
    Kevin Kho

    Kevin Kho

    4 months ago
    This is where the agent is spinning up many processes that don’t end right?
    So these processes are crowding the agent/server and causing Flows to get bottlenecked?
    Alexis Lucido

    Alexis Lucido

    4 months ago
    Exact Kevin!
    Except that now we have many processes related to docker, and less to pure Python
    Kevin Kho

    Kevin Kho

    4 months ago
    My responses will be a bit slow this week since I’m travelling, but I was thinking, maybe you can try a Docker agent instead so that the Flows run in a container and Docker cleans them? What you have is really not normal. You could also try the basic
    prefect server start
    and see if the behavior remains? I believe you used
    docker-compose
    right?
    Alexis Lucido

    Alexis Lucido

    4 months ago
    No worries Kevin! The problem initially observed was with prefect server start directly in the VM. I did use docker-compose
    I guess I'll give it a try with a Docker Agent! Thanks
    Kevin Kho

    Kevin Kho

    4 months ago
    Are you able to try Cloud just to see if the same processes are spun up?
    So we know if it’s an agent problem or server problem?
    Alexis Lucido

    Alexis Lucido

    4 months ago
    Kevin, sorry for my late answer, I was in vacation. I have been reflecting on your suggestions this morning and I guess I'll start with running Flows with a Docker agent. I'll keep you informed. Thank you very much