When you register your flows from code Python using e g `Mod Prefect Community #prefect-server

When you register your flows from code (Python), u...

Pierre Monico

08/26/2021, 8:40 AM

When you register your flows from code (Python), using e.g.

Module

storage with

stored_as_script=True

, why do all the flows and their dependencies need to be resolved? e.g. all my flows import

db

which is an instance of

sqlalchemy.create_engine()

that I setup with an environment variable. If I try to register e.g. from a CI pipeline, I have to set all those variables just to be able to register a script that will be stored as a file anyway? (NB: can’t use the CLI

prefect register

because it is not able to resolve any of the dependencies of my flows (they are in a folder), spitting

ModuleNotFoundError

)

Pierre Monico

08/26/2021, 8:45 AM

I might be missing something but after trying many approaches I find flow registration to be the weak spot of Prefect. On one hand the architecture is supposed to be decoupled with the flows using

Storage

, on the other hand you can register them directly from the

flow

object, thus de facto coupling the registration to your main repo (where your flow code lives). You need to specify a runner in which all dependencies can be resolved (makes sense ofc), but then

Storage

also needs all dependencies to be set up (at least when using

Module

). When trying to set it up “externally” - from the CLI - it won’t handle it because the flows are run as scripts, so if they are in a package they won’t be able to resolve any dependencies. Long story short: for now I didn’t find a straight forward way of deploying. But don’t get me wrong, still love Prefect 🙂

Zanie

08/26/2021, 2:03 PM

Hey Pierre, this is something we're working hard to address 🙂 but in the meantime, https://github.com/PrefectHQ/prefect/discussions/4042 may be helpful for reference

Zanie

08/26/2021, 2:04 PM

The flow's dependencies need to be resolved because we need to run the flow context to construct the DAG to send on registration

Zanie

08/26/2021, 2:05 PM

You can put the imports in a

Copy code

try:
  ...
except ModuleNotFoundError:
   print("Skipping module imports for registration...")

Pierre Monico

08/27/2021, 8:43 AM

Thanks @Zanie! I’m aware of that link and will definitely post my current setup there once it’s all polished. One more question: I also have a a bit of feedback regarding the docs; how can I contribute to those? Just send a PR? Or open a discussion first?

Zanie

08/27/2021, 3:20 PM

You can open a PR unless you're going to propose large changes then a discussion may be proper.

2 Views

Open in Slack

Previous Next