Matt Alhonte
05/24/2023, 1:42 AMprefect register <filepath>
in Prefect 1? Assume it's a bunch of py
files with Deployments defined in Python code.Marvin
05/24/2023, 1:43 AMpython
import os
import glob
from prefect import Client
client = Client()
# Replace 'your_directory' with the path to your directory containing the flows
for flow_file in glob.glob("your_directory/*.py"):
flow_module = os.path.splitext(os.path.basename(flow_file))[0]
flow = __import__(flow_module)
client.register(flow=flow.your_flow_name, project_name="your_project_name")
Replace your_directory
with the path to the directory containing your Python files with flows, and replace your_flow_name
and your_project_name
with the appropriate names. This script will register each flow in the specified directory.
Remember, the universe is a chaotic place, but at least we can bring some order to our flows.Matt Alhonte
05/24/2023, 1:44 AMNate
05/24/2023, 1:44 AMMatt Alhonte
05/24/2023, 1:45 AMprefect register <Flows folder>
Nate
05/24/2023, 1:46 AMMatt Alhonte
05/24/2023, 1:51 AMprefect register <Flows folder>
was run from the CLINate
05/24/2023, 1:54 AMMatt Alhonte
05/24/2023, 1:58 AM.py
file and 1 yml
file?Nate
05/24/2023, 2:13 PM1 flow would have to be 1not necessarily, you could have 1file and 1.py
file?yml
deployment.yaml
that specifies N deployments (in their own python files), as shown in the example I linked above
deployments:
- name: deployment-1
entrypoint: flows/hello.py:my_flow
parameters:
number: 42,
message: Don't panic!
work_pool:
name: my-process-work-pool
work_queue_name: primary-queue
- name: deployment-2
entrypoint: flows/goodbye.py:my_other_flow
work_pool:
name: my-process-work-pool
work_queue_name: secondary-queue
- name: deployment-3
entrypoint: flows/hello.py:yet_another_flow
work_pool:
name: my-docker-work-pool
work_queue_name: tertiary-queue
Matt Alhonte
05/24/2023, 8:04 PMpy
file? Minimally, it's Python file + a new entry in a yaml
file?py
file as a script one-by-one?alex
05/24/2023, 8:30 PMMatt Alhonte
05/24/2023, 8:32 PMyaml
or py
file telling the system about it") just seems like a lot more extra Friction & Negative Engineering.py
files one-by-one in another script will be an okay approximation, but I'm pretty sure it'll be a lot slower. We used to do it that way in our CI/CD pipeline, but when the prefect register <folder>
CLI command came out it shaved like 10 minutes off every run (which of course trickles down into much tighter feedback loops)if __name__ == "__main__":
memory = <number>
base_args = make_deployment_args(memory)
storage = S3Bucket.load(<name>)
deployment = Deployment.build_from_flow(
flow=log_flow,
parameters={"name": "Marvin"},
storage=storage,
**base_args
)
to the end of every Flow isn't THAT bad, but it's more boilerplate and less Declarative than it was in 1.0alex
05/24/2023, 8:46 PMMatt Alhonte
05/24/2023, 8:47 PMalex
05/24/2023, 8:55 PMMatt Alhonte
05/24/2023, 8:56 PM