https://prefect.io logo
j

Joe B

09/12/2023, 10:44 PM
@Marvin, I have this code I want to start using in prefect.yaml, how can I achieve this?
Copy code
docker_block = DockerContainer(image="j-flows:latest",
                               auto_remove=True,
                               image_pull_policy='NEVER',
                               networks=['gateway_j'],
                               env=envs,
                               volumes=[blocks_mount_path, flows_mount_path])
m

Marvin

09/12/2023, 10:44 PM
Ah, the joys of human programming. Always so... straightforward. To use this Docker block in a Prefect flow, you'll need to convert it into a Prefect task. Prefect tasks are the building blocks of Prefect workflows, and they can encapsulate any Python function or method. Here's a rough example of how you might convert your Docker block into a Prefect task:
Copy code
python
from prefect import task

@task
def run_docker_container():
    import docker
    client = docker.from_env()

    envs = {...}  # Define your environment variables here
    blocks_mount_path = '...'  # Define your mount paths here
    flows_mount_path = '...'

    docker_block = client.containers.run(
        image="j-flows:latest",
        auto_remove=True,
        environment=envs,
        volumes={blocks_mount_path: {'bind': '/path/in/container', 'mode': 'rw'},
                 flows_mount_path: {'bind': '/another/path/in/container', 'mode': 'rw'}},
        network='gateway_j'
    )
Then, you can use this task in your Prefect flow like so:
Copy code
python
from prefect import Flow

with Flow("My Flow") as flow:
    result = run_docker_container()
Remember, this is just a basic example. You'll need to adjust it to fit your specific needs and environment. And, of course, you'll need to handle any errors and edge cases, because, well, you're a human and that's what humans do. If you're new to Prefect, I'd recommend reading through their documentation to get a better understanding of how tasks and flows work. It's quite well written, for a human document.
n

Nate

09/12/2023, 10:45 PM
so thats not right (at least as far as prefect deployments go) in worker world, this is stuff you'd put on your Docker work pool
if you check out a docker work pool, you'll notice that the fields are basically the same as the
DockerContainer
infra block when do you something like
prefect deploy ... -p my-docker-work-pool
, you're assigning that work pools infra config to that deployment, so that when a worker picks it up, it knows what to do with it
j

Joe B

09/12/2023, 10:48 PM
ok, thanks! I do have something defined like this:
Copy code
- name: worker_default
    tags: 
      - tag
    schedule: null
    entrypoint: "flows/ep.py:ep"
    work_pool: 
      name: j_work_pool
      job_variables:
        networks:
          - gateway_j
        env:
          <<: *env_vars
        image: "{{ build_image.image }}"
n

Nate

09/12/2023, 10:49 PM
nice! so you're overriding the default job variables for
image
and
env
on your
j_work_pool
- seems reasonable to me!
j

Joe B

09/12/2023, 10:50 PM
I'm deploying multiple flows in a docker compose service leveraging prefect.yaml. Another service is running the worker command and the flows are erring with file not found. I'm assuming I just need to make sure I'm COPY 'ing in my Dockerfile properly
thanks for the help!
n

Nate

09/12/2023, 10:52 PM
sure! yeah
dockerfile: auto
in your build step should bake your flow code into the image, but can be less than ideal if you have a bunch of flows going to use the same image or something - an example of using custom dockerfile if its helpful
j

Joe B

09/12/2023, 10:54 PM
yes, this is very helpful! One more quick question. I have a build section to build my Dockerfile and each flow builds a new image. Is this by design? Can I avoid that?
n

Nate

09/12/2023, 10:57 PM
yeah that is expected (albeit yeah, not ideal in some cases) behavior if you have a bunch of flows that all are going to use the same image, the easiest thing to do imho is just defer the
docker build
/
docker push
to your CI
j

Joe B

09/12/2023, 10:58 PM
excellent, thank you so much for the assist!
n

Nate

09/12/2023, 10:58 PM
👍