https://prefect.io logo
s

Saurabh Indoria

01/27/2022, 5:01 AM
Hi all, In our prefect deployment, we see task logs like.. "Task finished with final status Pending". Does Lazarus process handle this? I have seen this stuck for hours.
k

Kevin Kho

01/27/2022, 5:10 AM
Are you using LocalDaskExecutor or DaskExecutor? Lazarus handles the events like not being able to start the Flow because of not being able to find hardware.
s

Saurabh Indoria

01/27/2022, 5:42 AM
We are using LocalDaskExecutor on Kubernetes Agent with Prefect Cloud
k

Kevin Kho

01/27/2022, 5:43 AM
Does this always happen or just sometimes?
s

Saurabh Indoria

01/27/2022, 5:44 AM
It happens sometimes... mostly when upstream task is stuck for a long time (which also happens sometimes), the downstream task will end up in Pending state and logs show Task finished with final status Pending
Even after the upstream task succeeds/fails, the downstream remains in Pending
k

Kevin Kho

01/27/2022, 5:50 AM
Do you have a similar setup to this ?
s

Saurabh Indoria

01/27/2022, 5:55 AM
Yes we do... Is there a way to resolve/improve this behaviour?
k

Kevin Kho

01/27/2022, 5:59 AM
It should resolve though eventually. Or does your Flow process just stop?
Does the
ClientError
or
manual_only
trigger fit what you are doing? Or just the
set_upstream
?
s

Saurabh Indoria

01/27/2022, 6:04 AM
I saw the Flow process just being stuck for hours. I had to manually restart the flow from Prefect UI.
We just use the set_upstream.. we use the functional API instead of the Imperative one
k

Kevin Kho

01/27/2022, 6:06 AM
With retries?
s

Saurabh Indoria

01/27/2022, 6:07 AM
Yes, all our tasks have default retries of 3 with 5 seconds delay
k

Kevin Kho

01/27/2022, 6:08 AM
Have you tried using
processes
instead of
threads
for LocalDaskExecutor?
s

Saurabh Indoria

01/27/2022, 6:08 AM
No, can you please point me to some documentation for that?
Also, the tasks that I debugged, for the upstream tasks which were stuck, I marked them (upstream) as failed from Prefect UI, and expected the downstream to work. Does setting the task state from Prefect UI affect the flow in anyway?
k

Kevin Kho

01/27/2022, 6:11 AM
Just try
LocalDaskExecutor(scheduler="processes")
and it might be more stable than threads. Yes setting from the UI will affect the Flow since the state is resolved already. So when a downstream checks the upstream dependencies, it will affect whether it starts or not.
🆗 1
s

Saurabh Indoria

01/27/2022, 6:12 AM
Sorry to bother you with this, but while we are on this topic, the upstream tasks which gets stuck are simple HTTP POST requests. Our API endpoint times out after 1 hour, so I assumed the task to fail or something after 1 hour, but I see it sometimes going on for hours. We use this task within
.map
so there are multiple parallel task runs. Some of these task runs would get stuck, like no useful logs other than task started running... Does this sound like a LocalDaskExecutor issue?
k

Kevin Kho

01/27/2022, 6:16 AM
No worries. I think this is an issue you will find with any executor then. It seems like there is nothing being returned. You can try a timeout on task level and see if it helps?
@task(timeout=timedelta(hours=1))
s

Saurabh Indoria

01/27/2022, 6:18 AM
Yes, I tried setting that up for all our tasks, but unfortunately that started messing up with one of our other tasks which is simple Pandas 1 liner code. Anyways, I'll set it again and see if I can fetch more details on what issue is causes. Thanks for your help @Kevin Kho!
k

Kevin Kho

01/27/2022, 6:21 AM
Of course!
6 Views