Hi, I have suddenly started to get this error on my flows using the Kubernetes Agent.

ERROR - prefect.CloudFlowRunner | Unexpected error: AttributeError("'Log' object has no attribute 'splitlines'")

I’m running prefect self hosted. Server, agent & flow versions are 0.14.22 Nothing has changed, any idea what could be the reason?

👋 Not sure I can help, but just trying to narrow down what this could be: • Is this recurring? Does it fail each time you run the flow? • Do you use Dask?

Hi, thanks for the reply.   - Yes it now happens every time I try and run my kuberenetes flows - Yes we do use Dask.  Here’s an example flow that fails

########## tasks ##########

@task()
def add_ten(x):
  print(f'starting {x}')
  sleep(10)
  print(f'ending {x}')
  return x * 2


########## run ##########

def kube_cluster():
  pod_spec = make_pod_spec(
    image="prefect:latest",
  )
  return KubeCluster(pod_spec)

with Flow("test1",
  executor=DaskExecutor(
  cluster_class=kube_cluster)
) as flow:
  t = add_ten(1)
  t1 = add_ten(2)
  t2 = add_ten(3)
  t3 = add_ten(10)
  t4 = add_ten(20)
  t5 = add_ten(30)

flow.run_config = KubernetesRun(
 image="prefect:latest"
)

My guess is

prefect:latest

OK thanks. I'll give it a try 🙂

I think you'll find that you've probably picked up a newer build that's broken something

Yep, that's working now. Thanks for your help

No worries! Glad it works ✅

Interesting, as far as I know we don't call

splitlines

in our repo so I'm not sure where this is happening. Can you share a longer traceback for the error or does it not exist?

@Zanie Have you perhaps pulled in a new Dask version in a new Prefect version? The only result I got when searching that error message led me to a page of Dask issues

I don't have a longer traceback, but I'm sure it was Dask related. I created a flow without dask and that still worked. We need to run prefect on centos based images so are using the github repo on python-centos:3.8 and installing the pip dependencies, so this might not be happening to anyone else. Once I updated the following commands in our custom Dockerfile to include version 0.14.22 it worked.

pip install --no-cache-dir <https://github.com/PrefectHQ/prefect.git@0.14.22#egg=prefect[all_orchestration_extras]>
git clone <https://github.com/PrefectHQ/prefect.git> --branch 0.14.22

I presume you're using Python 3.7+ which should always pull the latest Dask version (we only pin to an upper dask version for Python 3.6). I can look into this some more. What dask executor were you using?

Here’s the scripts I’m using. We’re running in Openshift too, hence the permission changes. I’m still quite new to this so may be some silly mistakes!

Python —version
Python 3.8.6

- custom prefect base image

FROM python-centos:3.8
USER root
ADD <https://github.com/krallin/tini/releases/download/v0.19.0/tini> /opt/app-root/bin/tini
RUN yum update -y && \
  pip install --upgrade pip && \
  yum install -y gcc git curl && \
  mkdir /root/.prefect/ && \
  pip install --no-cache-dir git+<https://github.com/PrefectHQ/prefect.git@0.14.22#egg=prefect[all_orchestration_extras]> && \
  pip install dask_kubernetes

RUN cd /tmp && \
  git clone <https://github.com/PrefectHQ/prefect.git> --branch 0.14.22 && \
  cp /tmp/prefect/entrypoint.sh /usr/local/bin/entrypoint.sh && \
  rm -rf /tmp/prefect

RUN yum remove -y git && \
  yum clean all && yum autoremove -y && \
  rm -rf /var/lib/apt/lists/*

RUN mkdir -p /.cache/pip /dask-worker-space /.local
RUN chgrp -R 0 /.local /.cache /usr /opt /dask-worker-space && \
  chmod -R g=u /.local /.cache /usr /opt /dask-worker-space && \
  chown -R 1001:0 /.local /.cache /usr /opt /dask-worker-space && \
  chmod 775 /.local /.cache /usr /opt /dask-worker-space && \
  chmod +x /opt/app-root/bin/tini

USER 1001
ENTRYPOINT ["tini", "-g", "--", "entrypoint.sh"]

- extract from flow

from prefect.executors import DaskExecutor
from dask_kubernetes import KubeCluster, make_pod_spec

########## run ##########

def kube_cluster():
  pod_spec = make_pod_spec(
    image="<custom prefect image>",
  )
  return KubeCluster(pod_spec)

with Flow("bb-parallel",
  executor=DaskExecutor(
  cluster_class=kube_cluster,
  adapt_kwargs={"minimum": 5, "maximum": 10})
) as flow:


flow.run_config = KubernetesRun(
 image="<custom prefect image>"
)

@Adam I don't think you need to explicitly install

dask_kubernetes

- Prefect does install a compatible version if you select

all_extras

I believe. It might not be included when you pick

all_orchestration_extras

- The

setup.py

is a bit confusing.

Ah ok, makes sense. I'll give that a try. Appreciate all the help on this 🙂

Haha let's count it as help if it works 🤣 I may very well be taking you down the wrong path