Pierre Monico
08/26/2021, 8:40 AMModule
storage with stored_as_script=True
, why do all the flows and their dependencies need to be resolved?
e.g. all my flows import db
which is an instance of sqlalchemy.create_engine()
that I setup with an environment variable. If I try to register e.g. from a CI pipeline, I have to set all those variables just to be able to register a script that will be stored as a file anyway?
(NB: can’t use the CLI prefect register
because it is not able to resolve any of the dependencies of my flows (they are in a folder), spitting ModuleNotFoundError
)Storage
, on the other hand you can register them directly from the flow
object, thus de facto coupling the registration to your main repo (where your flow code lives).
You need to specify a runner in which all dependencies can be resolved (makes sense ofc), but then Storage
also needs all dependencies to be set up (at least when using Module
).
When trying to set it up “externally” - from the CLI - it won’t handle it because the flows are run as scripts, so if they are in a package they won’t be able to resolve any dependencies.
Long story short: for now I didn’t find a straight forward way of deploying. But don’t get me wrong, still love Prefect 🙂Zanie
08/26/2021, 2:03 PMtry:
...
except ModuleNotFoundError:
print("Skipping module imports for registration...")
Pierre Monico
08/27/2021, 8:43 AMZanie
08/27/2021, 3:20 PM