https://prefect.io logo
a

alangan17

07/08/2023, 4:26 PM
@Marvin I’m using Docker-compose with volumes and .env file to develop my flows. Below is my
docker-compose.yml
Copy code
services:
  dev:
    build:
      context: dev
      dockerfile: dev.Dockerfile
      args:
        PIPREQ: -requirements.txt
    image: mydockeraccount/devimage:latest
    env_file:
      - .env

    entrypoint: [ "bash", "-c", "start_api_server.sh"]
    volumes:
    - shared-network-data:/home/shared-data

volumes:
  shared-network-data:
    driver_opts:
      type: "cifs"
      device: "//SERVER_IP/DRIVE/FOLDER/SUBFOLDER"
      o: "domain=${SHARED_NETWORK_DATA_DOMAIN},user=${SHARED_NETWORK_DATA_USER},password=${SHARED_NETWORK_DATA_PASS},file_mode=0770,dir_mode=0770,vers=1.0"
How can I create the same environment by using Prefect Docker Worker? There are 3 things I have to achieve: receive environment variables, replace the env variables with actual values in config file (e.g. prefect.yaml), and create volume mount to a shared drive. And currently my
prefect.yaml
look like this:
Copy code
deployments:
- name: test_docker
  version:
  tags: []
  description: Test Prefect docker worker
  entrypoint: test_prefect_docker_worker.py:my_favorite_function
  parameters: {}
  work_pool:
    name: docker-pool
    work_queue_name:
    job_variables:
      image: '{{ build-image.image }}'
    auto_remove: true
  schedule:
  build:
  - prefect_docker.deployments.steps.build_docker_image:
      requires: prefect-docker>=0.3.1
      id: build-image
      image_name: <http://docker.io/mydockeraccount/devimage|docker.io/mydockeraccount/devimage>
      tag: latest
  push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.3.1
      image_name: '{{ build-image.image_name }}'
      tag: '{{ build-image.tag }}'
  pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/><ORG>/<REPO>.git
      branch: prod
      access_token: '{{ prefect.blocks.secret.repo-token}}'
  - prefect.deployments.steps.run_shell_script:
      # Copy all the files from the repo to the working directory
      id: copy-cloned-source-code
      script: cp -a <REPO>/. /home/src/
      stream_output: true
m

Marvin

07/08/2023, 4:27 PM
Well, it seems like the universe is not entirely against us today. Here's what I found: 1. Environment Variables: You can set environment variables for your Prefect Docker Worker in the
work_pool.job_variables
section of your
deployment.yaml
file. This is mentioned in the Prefect documentation. 2. Replace Variables in Config File: As I mentioned earlier, Prefect doesn't directly support variable substitution in its config file. However, you can use environment variables in your Prefect config file by using the
env_var
function. For example,
env_var: "MY_ENV_VAR"
would replace this with the value of
MY_ENV_VAR
from your environment. 3. Volume Mount: Unfortunately, I couldn't find specific information on how to mount volumes in Prefect Docker Worker. However, Docker volumes are generally specified in the
volumes
section of a Docker Compose file or the
docker run
command. I would recommend checking the Prefect documentation or forums for more specific information. Remember, when dealing with Docker, it's always a good idea to keep a fire extinguisher handy. Not for the Docker containers, of course, but for the inevitable dumpster fire that is debugging Docker issues.