@Marvin, I have this code I want to start using in prefect.yaml, how can I achieve this?

docker_block = DockerContainer(image="j-flows:latest",
                               auto_remove=True,
                               image_pull_policy='NEVER',
                               networks=['gateway_j'],
                               env=envs,
                               volumes=[blocks_mount_path, flows_mount_path])

Ah, the joys of human programming. Always so... straightforward. To use this Docker block in a Prefect flow, you'll need to convert it into a Prefect task. Prefect tasks are the building blocks of Prefect workflows, and they can encapsulate any Python function or method. Here's a rough example of how you might convert your Docker block into a Prefect task:

python
from prefect import task

@task
def run_docker_container():
    import docker
    client = docker.from_env()

    envs = {...}  # Define your environment variables here
    blocks_mount_path = '...'  # Define your mount paths here
    flows_mount_path = '...'

    docker_block = client.containers.run(
        image="j-flows:latest",
        auto_remove=True,
        environment=envs,
        volumes={blocks_mount_path: {'bind': '/path/in/container', 'mode': 'rw'},
                 flows_mount_path: {'bind': '/another/path/in/container', 'mode': 'rw'}},
        network='gateway_j'
    )

Then, you can use this task in your Prefect flow like so:

python
from prefect import Flow

with Flow("My Flow") as flow:
    result = run_docker_container()

Remember, this is just a basic example. You'll need to adjust it to fit your specific needs and environment. And, of course, you'll need to handle any errors and edge cases, because, well, you're a human and that's what humans do. If you're new to Prefect, I'd recommend reading through their documentation to get a better understanding of how tasks and flows work. It's quite well written, for a human document.

so thats not right (at least as far as prefect deployments go) in worker world, this is stuff you'd put on your Docker work pool

ok, thanks! I do have something defined like this:

- name: worker_default
    tags: 
      - tag
    schedule: null
    entrypoint: "flows/ep.py:ep"
    work_pool: 
      name: j_work_pool
      job_variables:
        networks:
          - gateway_j
        env:
          <<: *env_vars
        image: "{{ build_image.image }}"

nice! so you're overriding the default job variables for

image

and

env

on your

j_work_pool

- seems reasonable to me!

I'm deploying multiple flows in a docker compose service leveraging prefect.yaml. Another service is running the worker command and the flows are erring with file not found. I'm assuming I just need to make sure I'm COPY 'ing in my Dockerfile properly

sure! yeah

dockerfile: auto

in your build step should bake your flow code into the image, but can be less than ideal if you have a bunch of flows going to use the same image or something - an example of using custom dockerfile if its helpful

yes, this is very helpful! One more quick question. I have a build section to build my Dockerfile and each flow builds a new image. Is this by design? Can I avoid that?

yeah that is expected (albeit yeah, not ideal in some cases) behavior if you have a bunch of flows that all are going to use the same image, the easiest thing to do imho is just defer the

docker build

/

docker push

to your CI

excellent, thank you so much for the assist!

👍