I'm having problems with tasks stuck at the "Running" state however when I inspected running workers everything seems running fine and results are submitted to my database although nothing is updating prefect state. currently using "prefect cloud" with

prefect 1.2.0-python3.9

docker image. same as https://github.com/PrefectHQ/prefect/issues/3952 for the dev team: https://cloud.prefect.io/bitthebyte/flow-run/b30223e1-5308-48fe-aa0b-9326c6e48860 (this is the stuck workflow) I already tried restarting

What is your executor? Dask or LocalDask?

Dask on Kubernetes

Would you be able to share your executor definition?

https://pastebin.com/D44uqfxR Password: SrmFhkJiNU Please note that I tested it before and everything was OK

Wow that’s the most complicated I’ve seen! Do you see anything unusual on the Dask dashboard?

I don't see anything I'd call unusual. but there is a warning regarding data size and mentioning the stuck task name. I don't think that is the cause though

UserWarning: Large object of size 4.47 MiB detected in task graph:                         ││   {'task': <Task: TaskName>, 'state': ... _parent': True}

I don’t think we can resolve this through Slack. I’ll talk to some people about this on Monday (the relevant ones are OOO) and then get back to you

False Alarm 😅 totally forgot I'm processing bulk data, not per task level which should only reflect if the whole bulk is finished

Oh I see