https://prefect.io logo
Title
t

Tomás Emilio Silva Ebensperger

12/07/2021, 4:01 PM
Hi everyone, so I have a flow that: 1. Runs fine when run locally 2. Runs fine when run manually on the server where I set up the the flows. 3. Runs fine when run manually on the prefect cloud (quick run) 4. DOES NOT WORK and gets stuck when run by the schedule set. I have explored the logs with debug level, I have tried everything. The flow just does not finish running. I have had issues with scheduling before. Please help.
k

Kevin Kho

12/07/2021, 4:09 PM
Hey @Tomás Emilio Silva Ebensperger, unclear what you mean here. So Flows triggered by a schedule run but get stuck in a
RUNNING
state? Do all of the tasks finish running but the Flow is left hanging?
t

Tomás Emilio Silva Ebensperger

12/07/2021, 4:29 PM
Hi Kevin, sorry for the confusion. It gets stuck at a certain task, but this only happens when the flow is run via a schedule run, never when i do quick run. the last message that shows on the logs is "Checking Flow Run State..."
k

Kevin Kho

12/07/2021, 4:36 PM
The task is somewhere in the middle? It’s kind of hard to guess what’s going on because the mechanism is the same for Quick Run. Quick Run creates a Flow with a schedule of the current time, which makes it get picked up immediately. Are there different parameters or RunConfig with the scheduled runs
t

Tomás Emilio Silva Ebensperger

12/07/2021, 4:39 PM
Yes, the task that gets stuck, is in the middle of other tasks. I do not see any differences in config between the scheduled way and the quick run option.
k

Kevin Kho

12/07/2021, 4:42 PM
Is that specific task compute intensive or does it take long?
And is it always the same task? How often do you see this with the scheduled runs? Every time? Sorry for all the questions. I just don’t have any immediate ideas. 😆. My current thought is do you have other flows at the same time that might be limiting the resources available?
e

Evan Sutherland

12/07/2021, 4:46 PM
possibly stupid question, but does the schedule have different parameters than the default?
t

Tomás Emilio Silva Ebensperger

12/07/2021, 4:46 PM
Yes, I think that might be it. Yes. It is a task somehow computationally expensive but I have run checks on the cpu and ram and it is all good. I just wonder why when run manually through quick run it always works. it is a flow that i run with other flows and sometimes you might have a few flows running at the same time. That task takes around 2 minutes when run successfully. toher tasks , where there is no issue, might take up to 5 minutes
k

Kevin Kho

12/07/2021, 4:49 PM
Yeah that sounds like this might be related to a lack of resources causing the FlowRunner to hang then. What is your agent? Local or Kubernetes?
t

Tomás Emilio Silva Ebensperger

12/07/2021, 5:00 PM
local
k

Kevin Kho

12/07/2021, 5:02 PM
I think it’s hard to say what behavior will be when resources are overwhelmed on Local. Python tends to just freeze and be slow.
t

Tomás Emilio Silva Ebensperger

12/07/2021, 5:03 PM
Ok Kevin, thank you. I will keep investigating.