Matthias
07/29/2022, 8:03 AMMathijs Carlu
07/29/2022, 9:07 AMMatthias
07/29/2022, 10:01 AMAnna Geller
where the flow file is baked into the image?it got so much better! you don't even have to bake your flow and even your custom modules into the image as they get packaged during deployment build the only thing that must be packaged are custom pip dependencies - example https://discourse.prefect.io/t/how-can-i-run-my-flow-in-a-docker-container/64#docker-deployment-using-a-custom-dockerfile-3
Mathijs Carlu
07/29/2022, 12:27 PMAnna Geller
Mathijs Carlu
07/29/2022, 1:26 PMLocalFileSystem
with path /app
2. built a deployment using prefect CLI
3. built a Docker image using the following docker file:
FROM prefecthq/prefect:2.0.0-python3.8
# Copy & install requirements
COPY requirements.txt /tmp/
RUN pip install -r /tmp/requirements.txt
COPY deployment.yaml /app/deployment.yaml
COPY etl-manifest.json /app/etl-manifest.json
COPY flow.py /app/flow.py
4. changed the following in deployment.yaml:
◦ image: image name from (3)
◦ storage: point at storage block created in (1)
This makes it possible to bake everything into 1 docker image, but I neglected the comments in the deployment.yaml file. Guess I'm kind of a bad boy 😎😝Anna Geller
whether it's already possible but not documented or if 'the recipe' has yet to be implemented.both actually I love that hack! 💙 And it's not a hack at all IMO - local file system is supposed to point to your flow code living in the execution env. If the execution env is a docker container, pointing to the local path on the container doesn't sound like a hack at all. Thanks so much @Mathijs Carlu, I'll include that in one of the recipes! 🙌
Mathijs Carlu
07/29/2022, 2:12 PM-sb local-file-system/docker-storage
(with docker-storage the path from (1) ). Now, this raises a NameError, because it looks for the /app path on my machine, which does not exist. Maybe this exception can be omitted in the case that infrastructure
is KubernetesJob
or DockerContainer
. This way, it wouldn't be necessary to modify the full storage part in the deployment.yaml fileAnna Geller
Jaime Raldua Veuthey
08/18/2022, 2:34 PMAnna Geller
Jaime Raldua Veuthey
08/18/2022, 2:39 PM