I have a very generic question about the Prefect architecture, the

Storage

in particular. My current project is using simple

flow.run()

s, all packaged in a Docker container. When switching to Prefect Server for orchestration, I have to register flows and thus define a

Storage

class,

RunConfig

etc. It seems like I have to write a lot of “infrastructure” code into my repo and reference different systems etc. I am not sure how to properly encapsulate this / separate these concerns from my business logic. I just want to be able to keep developing locally and use

flow.run

, but then have all the needed registration, storage configuration etc taken care of during deployment. I tried by simply adding a separate file in which I import the

flow

objects and then configure them before I register them, but e.g. with

Local()

storage and

save_as_script=True

, I then need to provide a path to the original flow file so I am not sure all the flow config made in the separate file will be taken into account. Long story short: I find the step from using core to orchestration very difficult to understand and to structure in terms of code - I have the feeling it’s kind of either-or. Positive closing: my experience with Prefect has been amazing so far 🙂

Hey @Pierre Monico, i think what you want is just a function that takes in a flow and sets up those pieces. Imagine this:

def setup_flow(flow):
    flow.storage = ...
    flow.run_config = ...
    return flow

with Flow(xxx) as flow:
     ...

flow = setup_flow(flow)

and then this setup flow can take in “local” or “S3" or something that lets you configure the output. you are right though that if you wanted to save this function somewhere and then import it, you would need to package it with your Flow with something like Docker as it would be a dependency. I think I know what you are saying with the

Local()

storage. It would be the path that you register from, but you need to package the dependencies together.

Thanks for the reply @Kevin Kho! I rather meant that I find it surprizing that I have to set up infrastructure from the flow directly rather than “externally”, but then found out you can also do it from the CLI. Also, this link helped a lot. I plan on posting my current setup there as soon as I get it done, along with a few points I found not-so-nice 😁 Overall the capabilities of Core + Server are really nice though, thanks for this!