alangan17
07/08/2023, 4:26 PMdocker-compose.yml
services:
dev:
build:
context: dev
dockerfile: dev.Dockerfile
args:
PIPREQ: -requirements.txt
image: mydockeraccount/devimage:latest
env_file:
- .env
entrypoint: [ "bash", "-c", "start_api_server.sh"]
volumes:
- shared-network-data:/home/shared-data
volumes:
shared-network-data:
driver_opts:
type: "cifs"
device: "//SERVER_IP/DRIVE/FOLDER/SUBFOLDER"
o: "domain=${SHARED_NETWORK_DATA_DOMAIN},user=${SHARED_NETWORK_DATA_USER},password=${SHARED_NETWORK_DATA_PASS},file_mode=0770,dir_mode=0770,vers=1.0"
How can I create the same environment by using Prefect Docker Worker? There are 3 things I have to achieve: receive environment variables, replace the env variables with actual values in config file (e.g. prefect.yaml), and create volume mount to a shared drive.
And currently my prefect.yaml
look like this:
deployments:
- name: test_docker
version:
tags: []
description: Test Prefect docker worker
entrypoint: test_prefect_docker_worker.py:my_favorite_function
parameters: {}
work_pool:
name: docker-pool
work_queue_name:
job_variables:
image: '{{ build-image.image }}'
auto_remove: true
schedule:
build:
- prefect_docker.deployments.steps.build_docker_image:
requires: prefect-docker>=0.3.1
id: build-image
image_name: <http://docker.io/mydockeraccount/devimage|docker.io/mydockeraccount/devimage>
tag: latest
push:
- prefect_docker.deployments.steps.push_docker_image:
requires: prefect-docker>=0.3.1
image_name: '{{ build-image.image_name }}'
tag: '{{ build-image.tag }}'
pull:
- prefect.deployments.steps.git_clone:
repository: <https://github.com/><ORG>/<REPO>.git
branch: prod
access_token: '{{ prefect.blocks.secret.repo-token}}'
- prefect.deployments.steps.run_shell_script:
# Copy all the files from the repo to the working directory
id: copy-cloned-source-code
script: cp -a <REPO>/. /home/src/
stream_output: true
Marvin
07/08/2023, 4:27 PMwork_pool.job_variables
section of your deployment.yaml
file. This is mentioned in the Prefect documentation.
2. Replace Variables in Config File: As I mentioned earlier, Prefect doesn't directly support variable substitution in its config file. However, you can use environment variables in your Prefect config file by using the env_var
function. For example, env_var: "MY_ENV_VAR"
would replace this with the value of MY_ENV_VAR
from your environment.
3. Volume Mount: Unfortunately, I couldn't find specific information on how to mount volumes in Prefect Docker Worker. However, Docker volumes are generally specified in the volumes
section of a Docker Compose file or the docker run
command. I would recommend checking the Prefect documentation or forums for more specific information.
Remember, when dealing with Docker, it's always a good idea to keep a fire extinguisher handy. Not for the Docker containers, of course, but for the inevitable dumpster fire that is debugging Docker issues.