Hi everyone, I am running prefect with docker with containers like agent, Orion etc. In the agent i have mentioned 2 queues. The problem is If i make a single inferencing request everything works file. But, if i trigger multiple inferencing request, queue goes unhealthy. Can anyone tell me what I am doing wrong here.

Are you using subflows?

Nope, just a normal flow, running sequentially

@flow(name="Inferencing")
def start_inferencing(my_param, file_path):
    print(">>>>>>>>>>>>>>>>>>>>>>>>>>>")
    print(my_param, file_path)
    print(">>>>>>>>>>>>>>>>>>>>>>>>>>>")
    res_1 = generate_jobID.submit(my_param)
    call_on_failure_if_failed(res_1)
    res_2 = get_file.submit(prev_task = file_path)
    call_on_failure_if_failed(res_2)
    res_3 = unzip.submit(prev_task = res_2)
    call_on_failure_if_failed(res_2)
    res_6 = add_pvt_tag.submit(prev_task = res_3)
    call_on_failure_if_failed(res_6)
    res_5 = send_data(prev_task = res_6)
    #call_on_failure_if_failed(res_5)


if __name__ == "main":
    start_inferencing()

Okay, got it. I was having very similar problems (agent crashing under any kind of load), and on top of that subflows were out of sync with the main flow and impossible to cancel. Switching to tasks made things a but better, but I guess this is not your situation

Correct, Basically I want to run multiple inferences concurrently (that is, multiple flows at the same time). I have one deployment with 1 queue assigned. How are you dealing with multiple request (if you are dealing).

Well, I am just hoping it doesn't crash, slowly planning to rewrite my pipleni in AWS step functions :) Btw, are you monitoring the CPU/memory load on the agent's server? It could he that launching multiple tasks exhausts all resources - memory could be the most likely culprit here

I did that too, but memory is not an issue here because the moment I trigger multiple (more than 1), queue state changes to unhealthy

I see, seems like a very specific problem

Yeah, i don't know if there's something worng with with flow and something else, hope someone from prefect will hop in here BTW, thanks for the help 🙂