Matthias
07/29/2022, 8:03 AMMathijs Carlu
07/29/2022, 9:07 AMMatthias
07/29/2022, 10:01 AMAnna Geller
07/29/2022, 12:04 PMwhere the flow file is baked into the image?it got so much better! you don't even have to bake your flow and even your custom modules into the image as they get packaged during deployment build the only thing that must be packaged are custom pip dependencies - example https://discourse.prefect.io/t/how-can-i-run-my-flow-in-a-docker-container/64#docker-deployment-using-a-custom-dockerfile-3
Mathijs Carlu
07/29/2022, 12:27 PMAnna Geller
07/29/2022, 1:03 PMMathijs Carlu
07/29/2022, 1:26 PMLocalFileSystem
with path /app
2. built a deployment using prefect CLI
3. built a Docker image using the following docker file:
FROM prefecthq/prefect:2.0.0-python3.8
# Copy & install requirements
COPY requirements.txt /tmp/
RUN pip install -r /tmp/requirements.txt
COPY deployment.yaml /app/deployment.yaml
COPY etl-manifest.json /app/etl-manifest.json
COPY flow.py /app/flow.py
4. changed the following in deployment.yaml:
◦ image: image name from (3)
◦ storage: point at storage block created in (1)
This makes it possible to bake everything into 1 docker image, but I neglected the comments in the deployment.yaml file. Guess I'm kind of a bad boy 😎😝Anna Geller
07/29/2022, 1:55 PMwhether it's already possible but not documented or if 'the recipe' has yet to be implemented.both actually I love that hack! 💙 And it's not a hack at all IMO - local file system is supposed to point to your flow code living in the execution env. If the execution env is a docker container, pointing to the local path on the container doesn't sound like a hack at all. Thanks so much @Mathijs Carlu, I'll include that in one of the recipes! 🙌
Mathijs Carlu
07/29/2022, 2:12 PM-sb local-file-system/docker-storage
(with docker-storage the path from (1) ). Now, this raises a NameError, because it looks for the /app path on my machine, which does not exist. Maybe this exception can be omitted in the case that infrastructure
is KubernetesJob
or DockerContainer
. This way, it wouldn't be necessary to modify the full storage part in the deployment.yaml fileAnna Geller
07/29/2022, 2:52 PMJaime Raldua Veuthey
08/18/2022, 2:34 PMAnna Geller
08/18/2022, 2:37 PMJaime Raldua Veuthey
08/18/2022, 2:39 PM