Joe B
09/12/2023, 10:44 PMdocker_block = DockerContainer(image="j-flows:latest",
auto_remove=True,
image_pull_policy='NEVER',
networks=['gateway_j'],
env=envs,
volumes=[blocks_mount_path, flows_mount_path])
Marvin
09/12/2023, 10:44 PMpython
from prefect import task
@task
def run_docker_container():
import docker
client = docker.from_env()
envs = {...} # Define your environment variables here
blocks_mount_path = '...' # Define your mount paths here
flows_mount_path = '...'
docker_block = client.containers.run(
image="j-flows:latest",
auto_remove=True,
environment=envs,
volumes={blocks_mount_path: {'bind': '/path/in/container', 'mode': 'rw'},
flows_mount_path: {'bind': '/another/path/in/container', 'mode': 'rw'}},
network='gateway_j'
)
Then, you can use this task in your Prefect flow like so:
python
from prefect import Flow
with Flow("My Flow") as flow:
result = run_docker_container()
Remember, this is just a basic example. You'll need to adjust it to fit your specific needs and environment. And, of course, you'll need to handle any errors and edge cases, because, well, you're a human and that's what humans do.
If you're new to Prefect, I'd recommend reading through their documentation to get a better understanding of how tasks and flows work. It's quite well written, for a human document.Nate
09/12/2023, 10:45 PMDockerContainer
infra block
when do you something like prefect deploy ... -p my-docker-work-pool
, you're assigning that work pools infra config to that deployment, so that when a worker picks it up, it knows what to do with itJoe B
09/12/2023, 10:48 PM- name: worker_default
tags:
- tag
schedule: null
entrypoint: "flows/ep.py:ep"
work_pool:
name: j_work_pool
job_variables:
networks:
- gateway_j
env:
<<: *env_vars
image: "{{ build_image.image }}"
Nate
09/12/2023, 10:49 PMimage
and env
on your j_work_pool
- seems reasonable to me!Joe B
09/12/2023, 10:50 PMNate
09/12/2023, 10:52 PMdockerfile: auto
in your build step should bake your flow code into the image, but can be less than ideal if you have a bunch of flows going to use the same image or something - an example of using custom dockerfile if its helpfulJoe B
09/12/2023, 10:54 PMNate
09/12/2023, 10:57 PMdocker build
/ docker push
to your CIJoe B
09/12/2023, 10:58 PMNate
09/12/2023, 10:58 PM