Wild question, but I can't seem to find anything definitive within the documentation. Is there a way to call prefect flows from within a docker image. I.E.: prefect-flow-1 calls prefect-flow-2 that exists in a docker container, but also contains prefect-flow-3 and prefect-flow-4.

hi @Derek - are you trying to execute flow runs from a deployment that are baked into an image? or just regular flow decorated python functions?

@Nate the former. The idea is to roll up groups of flows into monolithic docker containers vs. deploying updates to the prefect agent every time we modify a given flow.

what updates would you have to deploy to the prefect agent? baking all your deployment flows into an image is certainly possible, if that's what you need to do

Okay, so let me preface this all with "I'm dumb" so there's probably a better way to do this. But as it stands, I've got my normal flows baked into the actual prefect agent I'm deploying. Anytime I update a single flow, I have to push a new prefect agent. I'd rather just create some docker containers containing flows that get modified heavily and just pull those at the time of execution and execute the flow from the container.

hmm, yeah I would say it might be easier to just run an agent that can submit flow runs from a deployment as a docker container (i.e. DockerContainer infra), where all of those deployments cloud have the same infra block that specifies your monolithic

image

to use that way, when you change your code, you'll just have to rebuild the image being used by the deployments to include the new code, and not redeploy the agent ideally (in my general opinion) you have a base image that contains all common deps of your flows, but not the source itself, and then you clone the code just before runtime (that way you dont need to rebuild images just because your code changes) - lmk if you're interested in a pattern like that I can link some resources

@Nate yeah, definitely interested in a pattern like that. Appreciate any resources you can throw my way.

imo, this would be easiest with our new deployment ux

# Generic metadata about this project
name: prefect-monorepo
prefect-version: 2.11.1

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
    repository: <https://github.com/zzstoatzz/prefect-monorepo>
    branch: main
    access_token: null

# File for configuring project / deployment build, push and pull steps
definitions:
    work_pools:
        docker_work_pool: &docker_work_pool # set the image on this work pool in the UI 
            name: docker-work
            work_queue_name: default
            # or set it as a job_variable here
    schedules:
        every_ten_minutes: &every_10_minutes
            interval: 600

deployments:
  - name: healthcheck-demo
    entrypoint: src/demo_project/healthcheck.py:healthcheck
    schedule: *every_10_minutes
    parameters:
        message: Don't panic!
    work_pool: *docker_work_pool

you can have a

prefect.yaml

like this, which is responsible for defining deployments and mapping the deployments to schedules, infra, other config. and then I would add a simple github action (or similar) that builds your image for your repo with something like

name: Build and Push Docker Image

on:
  push:
    branches: [ main ]

  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Code
      uses: actions/checkout@v2
      
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v1

    - name: Login to DockerHub
      uses: docker/login-action@v1 
      with:
        username: ${{ secrets.DOCKERHUB_USERNAME }} # or wherever your images are going
        password: ${{ secrets.DOCKERHUB_PASSWORD }}
        
    - name: Build and push Docker image
      uses: docker/build-push-action@v2
      with:
        context: .
        file: path/to/your/Dockerfile
        push: true
        tags: username/repository:tag

How can I add my docker registry credentials into the prefect.yaml so that I can pull in an image from a private repo? I do not need to create an image on deployment, just pull it.