When you register your flows from code (Python), u...
# prefect-server
When you register your flows from code (Python), using e.g.
storage with
, why do all the flows and their dependencies need to be resolved? e.g. all my flows import
which is an instance of
that I setup with an environment variable. If I try to register e.g. from a CI pipeline, I have to set all those variables just to be able to register a script that will be stored as a file anyway? (NB: can’t use the CLI
prefect register
because it is not able to resolve any of the dependencies of my flows (they are in a folder), spitting
I might be missing something but after trying many approaches I find flow registration to be the weak spot of Prefect. On one hand the architecture is supposed to be decoupled with the flows using
, on the other hand you can register them directly from the
object, thus de facto coupling the registration to your main repo (where your flow code lives). You need to specify a runner in which all dependencies can be resolved (makes sense ofc), but then
also needs all dependencies to be set up (at least when using
). When trying to set it up “externally” - from the CLI - it won’t handle it because the flows are run as scripts, so if they are in a package they won’t be able to resolve any dependencies. Long story short: for now I didn’t find a straight forward way of deploying. But don’t get me wrong, still love Prefect 🙂
Hey Pierre, this is something we're working hard to address 🙂 but in the meantime, https://github.com/PrefectHQ/prefect/discussions/4042 may be helpful for reference
The flow's dependencies need to be resolved because we need to run the flow context to construct the DAG to send on registration
You can put the imports in a
Copy code
except ModuleNotFoundError:
   print("Skipping module imports for registration...")
Thanks @Zanie! I’m aware of that link and will definitely post my current setup there once it’s all polished. One more question: I also have a a bit of feedback regarding the docs; how can I contribute to those? Just send a PR? Or open a discussion first?
You can open a PR unless you're going to propose large changes then a discussion may be proper.