https://prefect.io logo
Title
i

Ilhom Hayot o'g'li

06/14/2022, 1:44 PM
Hi All! With prefect1.0 my parent flow stuck at status. I have given labels to it but still not working . Any suggestions on how to debug the flow? Status: It has been 15 seconds and your flow run has not been submitted by an agent. Agent 57f7d146-3361-4aef-b0e7-9961bb1fa183 (agent) has matching labels and last queried a few seconds ago. It should deploy your flow run.
k

Kevin Kho

06/14/2022, 1:47 PM
Looks like the agent is not able to pick up a Flow. Is your agent healthy?
i

Ilhom Hayot o'g'li

06/14/2022, 1:47 PM
I added each of my child flows to the agent but didnot add parent flows id prefect agent local start -l LAPTOP-ILKHOM -l dim_warehouse -l dim_company -l dim_customer_sold_to -l dim_batch -l dim_customer_invoice_ to
1
how to check it?
k

Kevin Kho

06/14/2022, 1:49 PM
I guess just check the logs whereever it’s running/
i

Ilhom Hayot o'g'li

06/14/2022, 1:57 PM
I think all my agents are not healthy
k

Kevin Kho

06/14/2022, 2:04 PM
Wait what? If you go to your local, can you check if the agent is alive?
i

Ilhom Hayot o'g'li

06/14/2022, 2:10 PM
I think it is still running no changes . I removed rest of the agents from prefect account, left only one. But still same error It has been 80 seconds and your flow run has not been submitted by an agent. Agent 57f7d146-3361-4aef-b0e7-9961bb1fa183 (agent) has matching labels and last queried a few seconds ago. It should deploy your flow run.
Is that because of the child flow as child flow will take some time
k

Kevin Kho

06/14/2022, 2:11 PM
Do you have more than 725 late flow runs by chance?
i

Ilhom Hayot o'g'li

06/14/2022, 2:12 PM
I'm sorry how to see/calculate late runs? is there some measurements/counters in the prefect account?
Oh I see it now. yes it comes out I have 746 late runs what does it mean? what to do?
k

Kevin Kho

06/14/2022, 2:29 PM
Yes exactly, you need to clear some because the scheduler freeezes at 725
There is a “Clear Late” button you can use there
i

Ilhom Hayot o'g'li

06/14/2022, 2:33 PM
late runs means - "my flow didnot manage to run all child flows until next cron time came". My child 1-2 min sometimes longer but usually under 2 mins. So I used 2 mins for cron. Is there any way of skipping late flows (only logging info about it)?
k

Kevin Kho

06/14/2022, 2:36 PM
Your scheduler will really not schedule new flows if you have more than 725 flow runs, but if you are asking about the Parent Flow skipping the child flow, maybe you are looking for something like this
i

Ilhom Hayot o'g'li

06/14/2022, 5:17 PM
I tried to add this portion of code to the parent flow but I have error. raise ClientError(result["errors"]) prefect.exceptions.ClientError: [{'path': ['flow_run'], 'message': 'invalid input syntax for type uuid: "parent_flow_example"', 'extensions': {'path': '$', 'code': 'data-exception', 'exception': {'message': 'invalid input syntax for type uuid: "parent_flow_example"'}}}]
k

Kevin Kho

06/14/2022, 5:18 PM
Can I see what you did?
i

Ilhom Hayot o'g'li

06/14/2022, 5:19 PM
sure
or should I implement it to the child flows?
k

Kevin Kho

06/14/2022, 5:22 PM
I think you want it on the child flow, not the parent
i

Ilhom Hayot o'g'li

06/14/2022, 5:24 PM
just 2 samples
Ok then you say I should skip the child if it is running. What I want to do is if child takes 3 min instead of 3 min skip it to the next round . If more than 4 rounds/loops missed warn me.
#some pseudocode what I am trying to do is run each 2 mins parent flowrun( child1_flow --- finished -then run it child2_flow - takes more than 2 mins ->then skip it to the next loop check all childs if some missed loop more than 4 time warn me )
k

Kevin Kho

06/14/2022, 5:29 PM
this is pretty complicated. could you elaborate more on the use case and we can see if there’s a better way to think about it?
i

Ilhom Hayot o'g'li

06/14/2022, 5:30 PM
it is for ETL processes to copy from one db to another database both in postgres
k

Kevin Kho

06/14/2022, 5:33 PM
so what does one child do?
i

Ilhom Hayot o'g'li

06/14/2022, 5:33 PM
both child has tasks smth like : check_difference -- return number--same for all childs extract_from_db1 -- return extracted data big transform --- preprocessing part/handling noises load_to_db2 -- load to the database in another server than the db2
k

Kevin Kho

06/14/2022, 6:32 PM
Sorry just got out of calls
Ok let me break this down: 1. Parent flow run makes sense. It kicks off multiple child flow runs (that I believe are different) 2. If a child takes more than 2 mins, skip it. This one is harder, it requires an Automation configured on the child flow to mark it as failed/cancelled if it’s taking too long. 3. If some missed the loop 4 times, send a warning - I am confused how many loops you are doing, but everything something fails, you can update the KVStore to keep a count, and if the KV store is above 4, you can send an alert. This KVStore activity can be done through a state handler. But the solution sounds massively complicated, and I think it can be easier if we reframe the problem. Let’s keep: 1. Parent flow runs multiple child flows 2. Maybe add Prefect timeouts to the child flow tasks and hope it works and they just raise Failed. 3. If a subflow fails, send a message with a state handler. I think the 4 failures makes things very complicated. What are you looping over? Is it a pre-defined list?