I'm having problems with tasks stuck at the "Runni...
# ask-community
a
I'm having problems with tasks stuck at the "Running" state however when I inspected running workers everything seems running fine and results are submitted to my database although nothing is updating prefect state. currently using "prefect cloud" with
prefect 1.2.0-python3.9
docker image. same as https://github.com/PrefectHQ/prefect/issues/3952 for the dev team: https://cloud.prefect.io/bitthebyte/flow-run/b30223e1-5308-48fe-aa0b-9326c6e48860 (this is the stuck workflow) I already tried restarting
k
What is your executor? Dask or LocalDask?
a
Dask on Kubernetes
k
Would you be able to share your executor definition?
a
https://pastebin.com/D44uqfxR Password: SrmFhkJiNU Please note that I tested it before and everything was OK
Tasks before were running and updating their state it is just the tail of the workflow that is stuck
k
Wow that’s the most complicated I’ve seen! Do you see anything unusual on the Dask dashboard?
a
I don't see anything I'd call unusual. but there is a warning regarding data size and mentioning the stuck task name. I don't think that is the cause though
Copy code
UserWarning: Large object of size 4.47 MiB detected in task graph:                         ││   {'task': <Task: TaskName>, 'state': ... _parent': True}
k
I don’t think we can resolve this through Slack. I’ll talk to some people about this on Monday (the relevant ones are OOO) and then get back to you
You can also add your details in the Github issue
a
False Alarm 😅 totally forgot I'm processing bulk data, not per task level which should only reflect if the whole bulk is finished
k
Oh I see
120 Views