Max Lei
03/01/2022, 7:23 PMDaskExecutor
, would DockerRun
be a good method to see if the dockerfile works or not?Anna Geller
03/01/2022, 7:30 PMflow.executor = DaskExecutor(
# FargateCluster is just one example
cluster_class="dask_cloudprovider.aws.FargateCluster",
cluster_kwargs={"n_workers": 4, "image": "your-prefect-image"},
)
Max Lei
03/01/2022, 7:36 PMAnna Geller
03/01/2022, 7:42 PMwith Flow(
"Dask Kubernetes Flow",
storage=storage,
executor=DaskExecutor(
cluster_class=lambda: KubeCluster(make_pod_spec(image=prefect.context.image)),
adapt_kwargs={"minimum": 2, "maximum": 3},
),
run_config=KubernetesRun(),
) as flow:
this blog post gives more info https://medium.com/slateco-blog/prefect-x-kubernetes-x-ephemeral-dask-power-without-responsibility-6e10b4f2fe40Kevin Kho
03/01/2022, 7:52 PMMax Lei
03/01/2022, 7:53 PMDockerRun
be the closest option?Kevin Kho
03/01/2022, 7:55 PMAnna Geller
03/01/2022, 7:57 PMdocker run --rm -it image_name
and then within the container run some code to test if everything has been properly installed and configured?Kevin Kho
03/01/2022, 8:00 PMAnna Geller
03/01/2022, 8:01 PMMax Lei
03/01/2022, 8:03 PMFailed to load and execute Flow's environment: ModuleNotFoundError("No module named '/home/ubuntu/'")
. Is this a docker container issue or something else?Anna Geller
03/01/2022, 8:06 PMKevin Kho
03/01/2022, 8:06 PMMax Lei
03/01/2022, 8:06 PMFROM prefecthq/prefect:0.11.4-python3.7
COPY ./dist/<PACKAGE_NAME_REMOVED>-0.0.0.tar.gz .
COPY requirements.txt .
COPY gh ./.config/
RUN apt update
RUN apt install -y curl
RUN curl -fsSL <https://cli.github.com/packages/githubcli-archive-keyring.gpg> | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg && \
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] <https://cli.github.com/packages> stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null
RUN apt update
RUN apt install -y build-essential gcc gh
RUN pip install setuptools==50.0.0
RUN pip install pip==20.2
RUN pip install pystan==2.17.1.0
RUN pip install fbprophet==0.7.1
RUN pip install -r requirements.txt
RUN pip install <PACKAGE_NAME_REMOVED>-0.0.0.tar.gz
WORKDIR /opt/prefect
COPY . /opt/prefect/flows/
DaskExecutor
?Anna Geller
03/01/2022, 8:08 PMKevin Kho
03/01/2022, 8:09 PMAnna Geller
03/01/2022, 8:09 PMMax Lei
03/01/2022, 8:10 PMDockerRun(image="<IMAGE_NAME_REMOVED>:latest")
, nothing special used in the flows.Kevin Kho
03/01/2022, 8:11 PMfbprophet
is so hard to deploy from experience. Like a pip install isn’t enough cuz it has some compiling afterMax Lei
03/01/2022, 8:12 PMfbprophet
is in requirements it would break, so you need to install it before your requirements.Kevin Kho
03/01/2022, 8:17 PMAnna Geller
03/01/2022, 8:17 PMKevin Kho
03/01/2022, 8:18 PMAnna Geller
03/01/2022, 8:19 PMMax Lei
03/01/2022, 8:20 PMMatthias
03/01/2022, 8:44 PM