a

    Ahmed Ezzat

    5 months ago
    I'm having problems with tasks stuck at the "Running" state however when I inspected running workers everything seems running fine and results are submitted to my database although nothing is updating prefect state. currently using "prefect cloud" with
    prefect 1.2.0-python3.9
    docker image. same as https://github.com/PrefectHQ/prefect/issues/3952 for the dev team: https://cloud.prefect.io/bitthebyte/flow-run/b30223e1-5308-48fe-aa0b-9326c6e48860 (this is the stuck workflow) I already tried restarting
    Kevin Kho

    Kevin Kho

    5 months ago
    What is your executor? Dask or LocalDask?
    a

    Ahmed Ezzat

    5 months ago
    Dask on Kubernetes
    Kevin Kho

    Kevin Kho

    5 months ago
    Would you be able to share your executor definition?
    a

    Ahmed Ezzat

    5 months ago
    https://pastebin.com/D44uqfxR Password: SrmFhkJiNU Please note that I tested it before and everything was OK
    Tasks before were running and updating their state it is just the tail of the workflow that is stuck
    Kevin Kho

    Kevin Kho

    5 months ago
    Wow that’s the most complicated I’ve seen! Do you see anything unusual on the Dask dashboard?
    a

    Ahmed Ezzat

    5 months ago
    I don't see anything I'd call unusual. but there is a warning regarding data size and mentioning the stuck task name. I don't think that is the cause though
    UserWarning: Large object of size 4.47 MiB detected in task graph:                         ││   {'task': <Task: TaskName>, 'state': ... _parent': True}
    Kevin Kho

    Kevin Kho

    5 months ago
    I don’t think we can resolve this through Slack. I’ll talk to some people about this on Monday (the relevant ones are OOO) and then get back to you
    You can also add your details in the Github issue
    a

    Ahmed Ezzat

    5 months ago
    False Alarm 😅 totally forgot I'm processing bulk data, not per task level which should only reflect if the whole bulk is finished
    Kevin Kho

    Kevin Kho

    5 months ago
    Oh I see