Hello everyone! During the past few weeks I have been learning how to use Prefect locally. However, now I am trying to deploy a flow using 4 OpenStack machines and I have some questions about the possible implementation: • Is it possible to make a flow of flows in which each flow executes on a specific machine of the 4 I have (without using dask)? Machines would share volumes between them. • In one of the machines I want to run a flow (triggered by other flow on a different machine) that must run a docker with a custom image when certain criteria is meet. Could I use the Docker Agent with the Local Executor for that flow? Thank you in advance! 🙏

Hi! This is definitely possible by leveraging different run configs for each child flow. And to define a specific condition, you can use the

case

task, as shown here:

from prefect import Flow, task, case
from prefect.tasks.prefect import create_flow_run, wait_for_flow_run
from prefect.run_configs import DockerRun


@task
def check_the_infrastructure():
    return "Docker with label machine-1"


with Flow("parent_flow") as flow:
    infra = check_the_infrastructure()
    with case(infra, "Docker with label machine-1"):
        child_flow_run_id = create_flow_run(
            flow_name="child_flow_name",
            run_config=DockerRun(
                labels=["machine-1"]
            ),  # with a specific condition like image
        )
        k8_child_flowrunview = wait_for_flow_run(
            child_flow_run_id, raise_final_state=True, stream_logs=True
        )
    with case(infra, "Docker with label machine-2"):
        child_flow_run_id = create_flow_run(
            flow_name="child_flow_name",
            run_config=DockerRun(
                labels=["machine-2"]
            ),  # with another different condition like image, label etc.
        )
        docker_child_flowrunview = wait_for_flow_run(
            child_flow_run_id, raise_final_state=True, stream_logs=True
        )

Thank you so much for your answer! I suppose that the configuration of the server and the machines should be as follows: • One of the machines acts as the local server, and the 4 machines should be register in the server (each one with a different label). This machine register the flows and make them run on the desired machine. • All machines should have Prefect installed and running with the same configuration. • The docker image should be in all the machines (or at least on the server and the one that is going to run the flow) Would this be enough for the communication between the 4 of them, or do I need something more?

When you say server, do you mean Prefect Server? The solution from above works the same way for both Prefect Cloud and Prefect Server. If so, then yes, Server can be running on one of those machines. Just make sure to set the Server endpoint in the

config.toml

on all agent instances: https://discourse.prefect.io/t/how-to-deploy-self-hosted-server-and-register-first-flows-to-the-server-backend/85 Also, that's correct - Prefect must be installed on all machines, otherwise you can't start an agent there. But they don't necessarily need to have the same configuration - they may be entirely different types of instances (one smaller one and another with much more CPU and memory) and they may (and should!) have different agent labels. Regarding the Docker image, it's best to push it to a remote container registry e.g. DockerHub, AWS ECR, GCP GCR, Azure AKS, etc. This way, when you change the image to include some new dependencies over time, your flows will be able to pull the new image without having to rewrite or redeploy your flow. The image can also be set dynamically on the run config (e.g. DockerRun) so that you may even override the default image for a specific flow at runtime.