Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
i
Ilhom Hayot o'g'li
06/14/2022, 1:44 PMHi All!
With prefect1.0 my parent flow stuck at status. I have given labels to it but still not working . Any suggestions on how to debug the flow?
Status:
It has been 15 seconds and your flow run has not been submitted by an agent. Agent 57f7d146-3361-4aef-b0e7-9961bb1fa183 (agent) has matching labels and last queried a few seconds ago. It should deploy your flow run.
k
Kevin Kho
06/14/2022, 1:47 PMLooks like the agent is not able to pick up a Flow. Is your agent healthy?
i
Ilhom Hayot o'g'li
06/14/2022, 1:47 PMI added each of my child flows to the agent but didnot add parent flows id
prefect agent local start -l LAPTOP-ILKHOM -l dim_warehouse -l dim_company -l dim_customer_sold_to -l dim_batch -l dim_customer_invoice_
to
✅ 1
how to check it?
k
Kevin Kho
06/14/2022, 1:49 PMI guess just check the logs whereever it’s running/
i
Ilhom Hayot o'g'li
06/14/2022, 1:57 PMI think all my agents are not healthy
k
Kevin Kho
06/14/2022, 2:04 PMWait what? If you go to your local, can you check if the agent is alive?
i
Ilhom Hayot o'g'li
06/14/2022, 2:10 PMI think it is still running no changes . I removed rest of the agents from prefect account, left only one. But still same error
It has been 80 seconds and your flow run has not been submitted by an agent. Agent 57f7d146-3361-4aef-b0e7-9961bb1fa183 (agent) has matching labels and last queried a few seconds ago. It should deploy your flow run.
Is that because of the child flow as child flow will take some time
k
Kevin Kho
06/14/2022, 2:11 PMDo you have more than 725 late flow runs by chance?
i
Ilhom Hayot o'g'li
06/14/2022, 2:12 PMI'm sorry how to see/calculate late runs? is there some measurements/counters in the prefect account?
Oh I see it now. yes it comes out I have 746 late runs what does it mean? what to do?
k
Kevin Kho
06/14/2022, 2:29 PMYes exactly, you need to clear some because the scheduler freeezes at 725
There is a “Clear Late” button you can use there
i
Ilhom Hayot o'g'li
06/14/2022, 2:33 PMlate runs means - "my flow didnot manage to run all child flows until next cron time came". My child 1-2 min sometimes longer but usually under 2 mins. So I used 2 mins for cron. Is there any way of skipping late flows (only logging info about it)?
k
Kevin Kho
06/14/2022, 2:36 PMYour scheduler will really not schedule new flows if you have more than 725 flow runs, but if you are asking about the Parent Flow skipping the child flow, maybe you are looking for something like this
i
Ilhom Hayot o'g'li
06/14/2022, 5:17 PMI tried to add this portion of code to the parent flow but I have error.
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['flow_run'], 'message': 'invalid input syntax for type uuid: "parent_flow_example"', 'extensions': {'path': '$', 'code': 'data-exception', 'exception': {'message': 'invalid input syntax for type uuid: "parent_flow_example"'}}}]
k
Kevin Kho
06/14/2022, 5:18 PMCan I see what you did?
i
Ilhom Hayot o'g'li
06/14/2022, 5:19 PMsure
or should I implement it to the child flows?
k
Kevin Kho
06/14/2022, 5:22 PMI think you want it on the child flow, not the parent
i
Ilhom Hayot o'g'li
06/14/2022, 5:24 PMjust 2 samples
Ok then you say I should skip the child if it is running. What I want to do is if child takes 3 min instead of 3 min skip it to the next round . If more than 4 rounds/loops missed warn me.
#some pseudocode what I am trying to do is
run each 2 mins
parent flowrun(
child1_flow --- finished -then run it
child2_flow - takes more than 2 mins ->then skip it to the next loop
check all childs if some missed loop more than 4 time warn me
)
k
Kevin Kho
06/14/2022, 5:29 PMthis is pretty complicated. could you elaborate more on the use case and we can see if there’s a better way to think about it?
i
Ilhom Hayot o'g'li
06/14/2022, 5:30 PMit is for ETL processes to copy from one db to another database both in postgres
k
Kevin Kho
06/14/2022, 5:33 PMso what does one child do?
i
Ilhom Hayot o'g'li
06/14/2022, 5:33 PMboth child has tasks smth like :
check_difference -- return number--same for all childs
extract_from_db1 -- return extracted data big
transform --- preprocessing part/handling noises
load_to_db2 -- load to the database in another server than the db2
k
Kevin Kho
06/14/2022, 6:32 PMSorry just got out of calls
Ok let me break this down:
1. Parent flow run makes sense. It kicks off multiple child flow runs (that I believe are different)
2. If a child takes more than 2 mins, skip it. This one is harder, it requires an Automation configured on the child flow to mark it as failed/cancelled if it’s taking too long.
3. If some missed the loop 4 times, send a warning - I am confused how many loops you are doing, but everything something fails, you can update the KVStore to keep a count, and if the KV store is above 4, you can send an alert. This KVStore activity can be done through a state handler.
But the solution sounds massively complicated, and I think it can be easier if we reframe the problem. Let’s keep:
1. Parent flow runs multiple child flows
2. Maybe add Prefect timeouts to the child flow tasks and hope it works and they just raise Failed.
3. If a subflow fails, send a message with a state handler. I think the 4 failures makes things very complicated.
What are you looping over? Is it a pre-defined list?