@Marvin Is there a simple way to register every single flow in a directory? The equivalent of

prefect register <filepath>

in Prefect 1? Assume it's a bunch of

py

files with Deployments defined in Python code.

In Prefect 2, you can register all flows in a directory by iterating through the Python files and registering each flow. Here's a simple Python script to achieve this:

python
import os
import glob
from prefect import Client

client = Client()

# Replace 'your_directory' with the path to your directory containing the flows
for flow_file in glob.glob("your_directory/*.py"):
    flow_module = os.path.splitext(os.path.basename(flow_file))[0]
    flow = __import__(flow_module)
    client.register(flow=flow.your_flow_name, project_name="your_project_name")

Replace

your_directory

with the path to the directory containing your Python files with flows, and replace

your_flow_name

and

your_project_name

with the appropriate names. This script will register each flow in the specified directory. Remember, the universe is a chaotic place, but at least we can bring some order to our flows.

oof, I was hoping that wasn't it, k

nope! marvin got that wrong

@Nate Projects don't look like they're quite ready for primetime, though?

yeah they’re still beta, but shouldnt undergo major interface changes. i like it personally, but yeah i totally get that

prefect register <Flows folder>

was run from the CLI

ah i misunderstood what you meant by “define everything in python” ill leave this here in case you wanna check it out https://docs.prefect.io/latest/concepts/projects/#working-with-multiple-deployments

@Nate So with that, 1 flow would have to be 1

.py

file and 1

yml

file?

1 flow would have to be 1

.py

file and 1

yml

file?

not necessarily, you could have 1

deployment.yaml

that specifies N deployments (in their own python files), as shown in the example I linked above

deployments:
  - name: deployment-1
    entrypoint: flows/hello.py:my_flow
    parameters:
        number: 42,
        message: Don't panic!
    work_pool:
        name: my-process-work-pool
        work_queue_name: primary-queue

  - name: deployment-2
    entrypoint: flows/goodbye.py:my_other_flow
    work_pool:
        name: my-process-work-pool
        work_queue_name: secondary-queue

  - name: deployment-3
    entrypoint: flows/hello.py:yet_another_flow
    work_pool:
        name: my-docker-work-pool
        work_queue_name: tertiary-queue

ack, so there's no way to just define everything in a

py

file? Minimally, it's Python file + a new entry in a

yaml

file?

Yeah, those are currently your two options for deploying lots of flows. Would you mind sharing what you enjoy about defining your deployments in Python or what causes you hesitation about defining deployments in YAML? As Nate mentioned, projects are still in beta, so any feedback you can provide would be super appreciated!

@alex So, deploying a new flow being a 1-step process ("just put new file in Flow folder") was wonderfully handy, and very much in keeping with the "Reduce/Eliminate Negative Engineering" philosophy that I like about Prefect

Thanks for sharing that context! Creating a similar experience in Prefect 2.0 will be tricky since deployments and flow are separate but related concepts, whereas, in 1.0, everything was bundled into the flow. How often do things like schedules and storage differ between your deployments?

Storage: Essentially never Schedules: Sometimes! Like we have flows that deal with different clients whose data is different sizes, where it can make sense to deploy with different-sized containers.

Gotcha, that’s doable, but as you said, you’ll need to write some additional Python or YAML to express that. Thanks for raising these friction points! We’ll have to think a little to come up with good ways to address them.

As long as it's within the same file, it's not a big deal. the Expensive friction points are Context Switches