https://prefect.io logo
n

Nimesh Kumar

02/10/2023, 9:07 AM
Hi everyone, I am running prefect with docker with containers like agent, Orion etc. In the agent i have mentioned 2 queues. The problem is If i make a single inferencing request everything works file. But, if i trigger multiple inferencing request, queue goes unhealthy. Can anyone tell me what I am doing wrong here.
t

Tadej Svetina

02/10/2023, 12:35 PM
Are you using subflows?
n

Nimesh Kumar

02/10/2023, 12:38 PM
Nope, just a normal flow, running sequentially
Copy code
@flow(name="Inferencing")
def start_inferencing(my_param, file_path):
    print(">>>>>>>>>>>>>>>>>>>>>>>>>>>")
    print(my_param, file_path)
    print(">>>>>>>>>>>>>>>>>>>>>>>>>>>")
    res_1 = generate_jobID.submit(my_param)
    call_on_failure_if_failed(res_1)
    res_2 = get_file.submit(prev_task = file_path)
    call_on_failure_if_failed(res_2)
    res_3 = unzip.submit(prev_task = res_2)
    call_on_failure_if_failed(res_2)
    res_6 = add_pvt_tag.submit(prev_task = res_3)
    call_on_failure_if_failed(res_6)
    res_5 = send_data(prev_task = res_6)
    #call_on_failure_if_failed(res_5)


if __name__ == "main":
    start_inferencing()
t

Tadej Svetina

02/10/2023, 12:40 PM
Okay, got it. I was having very similar problems (agent crashing under any kind of load), and on top of that subflows were out of sync with the main flow and impossible to cancel. Switching to tasks made things a but better, but I guess this is not your situation
n

Nimesh Kumar

02/10/2023, 12:44 PM
Correct, Basically I want to run multiple inferences concurrently (that is, multiple flows at the same time). I have one deployment with 1 queue assigned. How are you dealing with multiple request (if you are dealing).
t

Tadej Svetina

02/10/2023, 1:33 PM
Well, I am just hoping it doesn't crash, slowly planning to rewrite my pipleni in AWS step functions :) Btw, are you monitoring the CPU/memory load on the agent's server? It could he that launching multiple tasks exhausts all resources - memory could be the most likely culprit here
n

Nimesh Kumar

02/10/2023, 1:35 PM
I did that too, but memory is not an issue here because the moment I trigger multiple (more than 1), queue state changes to unhealthy
t

Tadej Svetina

02/10/2023, 1:37 PM
I see, seems like a very specific problem
n

Nimesh Kumar

02/10/2023, 1:38 PM
Yeah, i don't know if there's something worng with with flow and something else, hope someone from prefect will hop in here BTW, thanks for the help 🙂
🙏 1
Gentle reminder, can anyone please look into it