https://prefect.io logo
Title
o

Oliver Catterson

10/26/2021, 5:06 PM
Hi, We are have a odd issue where after around 30 minutes of a flow running, the arguments passed to a task appear as NoneTypes. "`"AttributeError: 'NoneType' object has no attribute 'start_date'""` We running prefect server 0.15.6 with the KubernetesRun run config and DaskExecutor. I have verifyed that the time is the issue by implementing a delay in a upstream task - without the delay the flow run completes successfully.
k

Kevin Kho

10/26/2021, 5:18 PM
Hey @Oliver Catterson, I think I need a bit more context here Is it a mapped task over Dask?
Do you have some skeleton code you can show me?
o

Oliver Catterson

10/26/2021, 5:38 PM
Yes it is a mapped task, All tasks after the 30 mins mark throw the NoneType error. I can provide some code if needed.
k

Kevin Kho

10/26/2021, 6:40 PM
Do you have Dask worker logs? I think what is potentially happening is that the workers are dying or autoscaling down and the data is being lost. Not super sure though.
h

haf

10/28/2021, 3:25 PM
I'm also seeing this error now (a bit randomly)
k

Kevin Kho

10/28/2021, 3:26 PM
Same question as above 😅. Are your workers dying somehow?
h

haf

10/28/2021, 3:26 PM
Not dask yet
Tried rerunning it, worked 🤷
Soon dask though
k

Kevin Kho

10/28/2021, 3:27 PM
Ah do you have a mapped task? It sounds like there is something failing silently in a task
h

haf

10/28/2021, 3:27 PM
A lot of mapped tasks
k

Kevin Kho

10/28/2021, 3:28 PM
Is this specifically happening in one of those? Does the task upstream of that fail intermittently?