https://prefect.io logo
Title
t

Thomas Opsomer

03/08/2022, 2:55 PM
Hello 🙂 We have issues with tasks that require a manual approval (with
manual_only
trigger), after being approved using the UI on prefect cloud, it takes a very long time before the flow is resumed (like several hours :/). It's not the first time it happens, but currently it's happening on every flows. Anyone having the same issue ? Is it an issue on prefect backend side or should we do something ?
k

Kevin Kho

03/08/2022, 2:57 PM
Oh man that is the second report. We’ll look into it
👍 1
Could you give me flow run ids?
t

Thomas Opsomer

03/08/2022, 3:06 PM
this one should be stuck: 59b5eec1-5139-4dc4-b429-998eb852d7e5
k

Kevin Kho

03/08/2022, 3:06 PM
Thanks!
t

Thomas Opsomer

03/08/2022, 3:06 PM
another one 9f15dc61-e502-4267-8ba0-1633a882c73c
I'll have to resume these flows ^^ Do you want to me to keep them stuck a bit longer for investigation ?
k

Kevin Kho

03/08/2022, 5:05 PM
Can you resume them? We have a clue what is wrong. There is a backend service not working correctly and we are working on a fix but do whatever you need to, I was able to replicate and have flow runs stuck in the Resume state as well
t

Thomas Opsomer

03/08/2022, 5:09 PM
Yes, in our case these tasks are just "waiting" tasks, they don't do anything, so I just set them as success and then restart the flow...
k

Kevin Kho

03/08/2022, 5:10 PM
Ah ok go for it. I was wondering if you had a hack to get it to execute
Hey @Thomas Opsomer, we found an API query that was timing out in Cloud for the service that handles resuming these tasks. We made edits to it and deployed it and the new deployment just went through but it is still timing out so this is not resolved for now and we are still working on a fix. Sorry about that. (edited)
This has been resolved so new flows will work and then the old ones are being processed now
t

Thomas Opsomer

03/11/2022, 4:49 PM
I confirm it works fine ! Thanks
k

Kevin Kho

03/11/2022, 4:55 PM
uhh the other guy said 15% of flows fail. how many do you have I am just curious?
t

Thomas Opsomer

03/11/2022, 4:59 PM
We don't have much flows running right now. One of my colleague said it worked for the flow he was testing. Our production flows run sunday/monday so I'll give you better feedback next week 🙂
k

Kevin Kho

03/11/2022, 5:01 PM
ok thanks!
t

Thomas Opsomer

03/11/2022, 5:16 PM
Is this feature used a lot by other users ? I would think that more users would report issues, but didn't find much here. Also I read somewhere that it may not be in the v2, so I'm curious
@Kevin Kho to follow up, I still have some flow stuck on tasks with manual validation 😕
here is a flow_id: "c31b77e9-af98-4ac3-a6df-987c9e667d03" if it's of any use 🙂
k

Kevin Kho

03/14/2022, 5:08 PM
Thanks will bring this up. What percent suffer from this and how long is it taking?
t

Thomas Opsomer

03/14/2022, 5:32 PM
ah sorry actually it's a flow that was started last week
k

Kevin Kho

03/14/2022, 5:36 PM
Ah ok