https://prefect.io logo
Title
a

Ahmed Ezzat

04/15/2022, 5:29 PM
I'm having problems with tasks stuck at the "Running" state however when I inspected running workers everything seems running fine and results are submitted to my database although nothing is updating prefect state. currently using "prefect cloud" with
prefect 1.2.0-python3.9
docker image. same as https://github.com/PrefectHQ/prefect/issues/3952 for the dev team: https://cloud.prefect.io/bitthebyte/flow-run/b30223e1-5308-48fe-aa0b-9326c6e48860 (this is the stuck workflow) I already tried restarting
k

Kevin Kho

04/15/2022, 5:31 PM
What is your executor? Dask or LocalDask?
a

Ahmed Ezzat

04/15/2022, 5:32 PM
Dask on Kubernetes
k

Kevin Kho

04/15/2022, 5:33 PM
Would you be able to share your executor definition?
a

Ahmed Ezzat

04/15/2022, 5:35 PM
https://pastebin.com/D44uqfxR Password: SrmFhkJiNU Please note that I tested it before and everything was OK
Tasks before were running and updating their state it is just the tail of the workflow that is stuck
k

Kevin Kho

04/15/2022, 5:37 PM
Wow that’s the most complicated I’ve seen! Do you see anything unusual on the Dask dashboard?
a

Ahmed Ezzat

04/15/2022, 5:47 PM
I don't see anything I'd call unusual. but there is a warning regarding data size and mentioning the stuck task name. I don't think that is the cause though
UserWarning: Large object of size 4.47 MiB detected in task graph:                         ││   {'task': <Task: TaskName>, 'state': ... _parent': True}
k

Kevin Kho

04/15/2022, 5:50 PM
I don’t think we can resolve this through Slack. I’ll talk to some people about this on Monday (the relevant ones are OOO) and then get back to you
You can also add your details in the Github issue
a

Ahmed Ezzat

04/15/2022, 6:55 PM
False Alarm 😅 totally forgot I'm processing bulk data, not per task level which should only reflect if the whole bulk is finished
k

Kevin Kho

04/15/2022, 6:57 PM
Oh I see