https://prefect.io logo
Title
p

Pierre Monico

08/05/2021, 12:55 PM
I have a very generic question about the Prefect architecture, the
Storage
in particular. My current project is using simple
flow.run()
s, all packaged in a Docker container. When switching to Prefect Server for orchestration, I have to register flows and thus define a
Storage
class,
RunConfig
etc. It seems like I have to write a lot of “infrastructure” code into my repo and reference different systems etc. I am not sure how to properly encapsulate this / separate these concerns from my business logic. I just want to be able to keep developing locally and use
flow.run
, but then have all the needed registration, storage configuration etc taken care of during deployment. I tried by simply adding a separate file in which I import the
flow
objects and then configure them before I register them, but e.g. with
Local()
storage and
save_as_script=True
, I then need to provide a path to the original flow file so I am not sure all the flow config made in the separate file will be taken into account. Long story short: I find the step from using core to orchestration very difficult to understand and to structure in terms of code - I have the feeling it’s kind of either-or. Positive closing: my experience with Prefect has been amazing so far 🙂
k

Kevin Kho

08/05/2021, 3:06 PM
Hey @Pierre Monico, i think what you want is just a function that takes in a flow and sets up those pieces. Imagine this:
def setup_flow(flow):
    flow.storage = ...
    flow.run_config = ...
    return flow

with Flow(xxx) as flow:
     ...

flow = setup_flow(flow)
and then this setup flow can take in “local” or “S3" or something that lets you configure the output. you are right though that if you wanted to save this function somewhere and then import it, you would need to package it with your Flow with something like Docker as it would be a dependency. I think I know what you are saying with the
Local()
storage. It would be the path that you register from, but you need to package the dependencies together.
p

Pierre Monico

08/09/2021, 12:09 PM
Thanks for the reply @Kevin Kho! I rather meant that I find it surprizing that I have to set up infrastructure from the flow directly rather than “externally”, but then found out you can also do it from the CLI. Also, this link helped a lot. I plan on posting my current setup there as soon as I get it done, along with a few points I found not-so-nice 😁 Overall the capabilities of Core + Server are really nice though, thanks for this!
👍 1