Oliver Catterson

    Oliver Catterson

    10 months ago
    Hi, We are have a odd issue where after around 30 minutes of a flow running, the arguments passed to a task appear as NoneTypes. ""AttributeError: 'NoneType' object has no attribute 'start_date'"" We running prefect server 0.15.6 with the KubernetesRun run config and DaskExecutor. I have verifyed that the time is the issue by implementing a delay in a upstream task - without the delay the flow run completes successfully.
    Kevin Kho

    Kevin Kho

    10 months ago
    Hey @Oliver Catterson, I think I need a bit more context here Is it a mapped task over Dask?
    Do you have some skeleton code you can show me?
    Oliver Catterson

    Oliver Catterson

    10 months ago
    Yes it is a mapped task, All tasks after the 30 mins mark throw the NoneType error. I can provide some code if needed.
    Kevin Kho

    Kevin Kho

    10 months ago
    Do you have Dask worker logs? I think what is potentially happening is that the workers are dying or autoscaling down and the data is being lost. Not super sure though.
    haf

    haf

    10 months ago
    I'm also seeing this error now (a bit randomly)
    Kevin Kho

    Kevin Kho

    10 months ago
    Same question as above 😅. Are your workers dying somehow?
    haf

    haf

    10 months ago
    Not dask yet
    Tried rerunning it, worked 🤷
    Soon dask though
    Kevin Kho

    Kevin Kho

    10 months ago
    Ah do you have a mapped task? It sounds like there is something failing silently in a task
    haf

    haf

    10 months ago
    A lot of mapped tasks
    Kevin Kho

    Kevin Kho

    10 months ago
    Is this specifically happening in one of those? Does the task upstream of that fail intermittently?