Pierre Monico
08/05/2021, 12:55 PMStorage
in particular.
My current project is using simple flow.run()
s, all packaged in a Docker container. When switching to Prefect Server for orchestration, I have to register flows and thus define a Storage
class, RunConfig
etc. It seems like I have to write a lot of “infrastructure” code into my repo and reference different systems etc. I am not sure how to properly encapsulate this / separate these concerns from my business logic.
I just want to be able to keep developing locally and use flow.run
, but then have all the needed registration, storage configuration etc taken care of during deployment.
I tried by simply adding a separate file in which I import the flow
objects and then configure them before I register them, but e.g. with Local()
storage and save_as_script=True
, I then need to provide a path to the original flow file so I am not sure all the flow config made in the separate file will be taken into account.
Long story short: I find the step from using core to orchestration very difficult to understand and to structure in terms of code - I have the feeling it’s kind of either-or.
Positive closing: my experience with Prefect has been amazing so far 🙂Kevin Kho
08/05/2021, 3:06 PMdef setup_flow(flow):
flow.storage = ...
flow.run_config = ...
return flow
with Flow(xxx) as flow:
...
flow = setup_flow(flow)
and then this setup flow can take in “local” or “S3" or something that lets you configure the output.
you are right though that if you wanted to save this function somewhere and then import it, you would need to package it with your Flow with something like Docker as it would be a dependency.
I think I know what you are saying with the Local()
storage. It would be the path that you register from, but you need to package the dependencies together.Pierre Monico
08/09/2021, 12:09 PM