Riley Hun
08/12/2020, 4:33 PMprefecthq/prefect:0.12.1-python3.7
and prefecthq/prefect:0.12.5-python3.7
, which resulted in the following error:
AttributeError: module 'types' has no attribute 'CellType'
I've also tried FROM prefecthq/prefect:0.11.5-python3.7
, which resulted in this error:
ModuleNotFoundError: No module named 'prefect.core.parameter'
FROM prefecthq/prefect:0.12.1-python3.7
ADD ./alternative_data_pipelines /root/lib/alternative_data_pipelines
ADD key.json /creds/key.json
ADD feature_lib_creds.json /creds/feature_lib_creds.json
RUN pip install /root/lib/alternative_data_pipelines
ENV PYTHONPATH ${PYTHONPATH}:/root/lib/
RUN apt-get update \
&& apt-get -qq -y install curl
# Downloading gcloud package
RUN curl <https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz> > /tmp/google-cloud-sdk.tar.gz
# Installing the package
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh
# Adding the package path to local
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
Chris White
08/12/2020, 4:35 PMRiley Hun
08/12/2020, 4:36 PMjames.lamb
08/12/2020, 9:24 PMWebhook
⢠agent: KubernetesAgent
⢠environment: KubernetesJobEnvironment
⢠executor: DaskExecutor
⢠prefect version: 0.13.2
And I got this same error during the job created by KubernetesAgent
to run a flow.
This issue (https://github.com/mlflow/mlflow/issues/2429) makes me think that the problem is incompatible versions of cloudpickle
between the environment where flow.register(build=True)
was called and the one where the flow runs.
I have cloudpickle
1.5.0 in the environment where I registered the flow and cloudpickle
1.4.1 in the image where it's being run.
This is extra tough because the most recent Dask distributed
has a floor of 1.5.0
ERROR: distributed 2.22.0 has requirement cloudpickle>=1.5.0, but you'll have cloudpickle 1.4.1 which is incompatible.
I tried downgrading cloudpickle
to 1.4.1 and re-registering my flow (with build=True
), and then re-ran it. The new run was successful.The erroris the result of different minor versions ofAttributeError: module 'types' has no attribute 'CellType'
between the environment where a flow runs and the environment where it was registered.cloudpickle
Chris White
08/12/2020, 9:35 PMMarvin
08/12/2020, 9:35 PM