David Ojeda
05/07/2020, 2:53 PMDaskKubernetesEnvironment
and a kubernetes agent on a k8s GCP cluster that has auto-scaling (I am not sure if this is relevant).
The strange situation is that, sometimes, the flow stays on running indefinetely because the last 1 to 3 tasks of the map are “still running”. I will try to collect some logs when this happens, but it does not look like the workers are stuck; it seems more like they were killed a bit too soon and they could not inform the prefect server (or the dask scheduler?) that they finished…
Has anyone encountered this strange situation?Laura Lorenz (she/her)
05/07/2020, 3:04 PMDavid Ojeda
05/07/2020, 3:08 PMLaura Lorenz (she/her)
05/07/2020, 3:25 PMDavid Ojeda
05/08/2020, 9:20 AMwork_stealing=True
on my DaskKubernetesEnvironment
? I will remove this and try again"Task 'xxx': task is already running"
)Laura Lorenz (she/her)
05/09/2020, 2:55 PMDavid Ojeda
05/11/2020, 6:43 AM