https://prefect.io logo
Join Slack
Powered by
Hi All! With prefect1.0 my parent flow stuck at st...
# ask-community
i
Hi All! With prefect1.0 my parent flow stuck at status. I have given labels to it but still not working . Any suggestions on how to debug the flow? Status: It has been 15 seconds and your flow run has not been submitted by an agent. Agent 57f7d146-3361-4aef-b0e7-9961bb1fa183 (agent) has matching labels and last queried a few seconds ago. It should deploy your flow run.
k
Looks like the agent is not able to pick up a Flow. Is your agent healthy?
i
I added each of my child flows to the agent but didnot add parent flows id prefect agent local start -l LAPTOP-ILKHOM -l dim_warehouse -l dim_company -l dim_customer_sold_to -l dim_batch -l dim_customer_invoice_ to
✅ 1
how to check it?
k
I guess just check the logs whereever it’s running/
i
I think all my agents are not healthy
k
Wait what? If you go to your local, can you check if the agent is alive?
i
I think it is still running no changes . I removed rest of the agents from prefect account, left only one. But still same error It has been 80 seconds and your flow run has not been submitted by an agent. Agent 57f7d146-3361-4aef-b0e7-9961bb1fa183 (agent) has matching labels and last queried a few seconds ago. It should deploy your flow run.
Is that because of the child flow as child flow will take some time
k
Do you have more than 725 late flow runs by chance?
i
I'm sorry how to see/calculate late runs? is there some measurements/counters in the prefect account?
Oh I see it now. yes it comes out I have 746 late runs what does it mean? what to do?
k
Yes exactly, you need to clear some because the scheduler freeezes at 725
There is a “Clear Late” button you can use there
i
late runs means - "my flow didnot manage to run all child flows until next cron time came". My child 1-2 min sometimes longer but usually under 2 mins. So I used 2 mins for cron. Is there any way of skipping late flows (only logging info about it)?
k
Your scheduler will really not schedule new flows if you have more than 725 flow runs, but if you are asking about the Parent Flow skipping the child flow, maybe you are looking for something like this
i
I tried to add this portion of code to the parent flow but I have error. raise ClientError(result["errors"]) prefect.exceptions.ClientError: [{'path': ['flow_run'], 'message': 'invalid input syntax for type uuid: "parent_flow_example"', 'extensions': {'path': '$', 'code': 'data-exception', 'exception': {'message': 'invalid input syntax for type uuid: "parent_flow_example"'}}}]
k
Can I see what you did?
i
sure
or should I implement it to the child flows?
k
I think you want it on the child flow, not the parent
i
just 2 samples
Ok then you say I should skip the child if it is running. What I want to do is if child takes 3 min instead of 3 min skip it to the next round . If more than 4 rounds/loops missed warn me.
#some pseudocode what I am trying to do is run each 2 mins parent flowrun( child1_flow --- finished -then run it child2_flow - takes more than 2 mins ->then skip it to the next loop check all childs if some missed loop more than 4 time warn me )
k
this is pretty complicated. could you elaborate more on the use case and we can see if there’s a better way to think about it?
i
it is for ETL processes to copy from one db to another database both in postgres
k
so what does one child do?
i
both child has tasks smth like : check_difference -- return number--same for all childs extract_from_db1 -- return extracted data big transform --- preprocessing part/handling noises load_to_db2 -- load to the database in another server than the db2
k
Sorry just got out of calls
Ok let me break this down: 1. Parent flow run makes sense. It kicks off multiple child flow runs (that I believe are different) 2. If a child takes more than 2 mins, skip it. This one is harder, it requires an Automation configured on the child flow to mark it as failed/cancelled if it’s taking too long. 3. If some missed the loop 4 times, send a warning - I am confused how many loops you are doing, but everything something fails, you can update the KVStore to keep a count, and if the KV store is above 4, you can send an alert. This KVStore activity can be done through a state handler. But the solution sounds massively complicated, and I think it can be easier if we reframe the problem. Let’s keep: 1. Parent flow runs multiple child flows 2. Maybe add Prefect timeouts to the child flow tasks and hope it works and they just raise Failed. 3. If a subflow fails, send a message with a state handler. I think the 4 failures makes things very complicated. What are you looping over? Is it a pre-defined list?
13 Views
Open in Slack
PreviousNext
Powered by