Hey there, looking for some ideation. My team has set up a monolithic repo for our various data rel...

Mitchell Bregman

10/21/2020, 4:32 PM

Hey there, looking for some ideation. My team has set up a monolithic repo for our various data related flows, attached with a CircleCI workflow to handle registering to Prefect Cloud. I was wondering - has anyone set up a CI/CD with multiple users? How were you able to register flows for a given user via CI/CD? Have people created service-level User accounts that consolidate all flow registration? Thanks!

Zanie

10/21/2020, 5:06 PM

Hi @Mitchell Bregman

Zanie

10/21/2020, 5:07 PM

How are the flows associated with a CI run? Is a commit going to be associated with the flows to register? Are the flows grouped in your code by user?

Mitchell Bregman

10/21/2020, 5:13 PM

upon merge from

develop

master

, we run a workflow that registers all the flows to the cloud… what really happens under the hood is - we’ve build an abstraction around prefect flows that configure all of our storage/environment/helper needs.. code is grouped by relevant data source, i.e.

snowflake, mysql, postgres

Mitchell Bregman

10/21/2020, 5:14 PM

does that make sense? — it is not grouped by user. did i answer the commit associated question?

Zanie

10/21/2020, 5:18 PM

Yeah that makes sense! So how are your users split up? What associates flows with a user and what ‘user’ are you talking about here?

How were you able to register flows for a given user via CI/CD?

Mitchell Bregman

10/21/2020, 5:26 PM

So we have a bunch of user accounts that have read/write perms. I just wanted to see if there was an idea of how I can associate a flow that a user built (perhaps a username string or something) to their cloud key…. the final function for registering currently looks like this for me:

Copy code

for flow_cls in ParkMobileFlow.__subclasses__():
        flow = flow_cls()
        flow.register()

Mitchell Bregman

10/21/2020, 5:26 PM

i dont think this is the most ideal ^

Mitchell Bregman

10/21/2020, 5:26 PM

any thoughts?

Marko Herkaliuk

10/21/2020, 5:30 PM

maybe some like DAG owner in Airflow?

Mitchell Bregman

10/21/2020, 5:30 PM

yes ^^

Zanie

10/21/2020, 5:31 PM

I think I get what you mean now, I’ll look into what’s feasible

Zanie

10/21/2020, 5:55 PM

After looking into this a bit more, you could do a pattern where each flow has an in-code tag identifying what user wrote it and we could add support to

flow.register()

to take an auth token which you would provide by peeking at the user for the flow. However, this requires managing multiple tokens in a safe way and introduces complexity for our handling of client tokens which would affect much more than just

flow.register

. The way it’s implemented currently we have a clear opinion that if a client authenticates correctly in one place in your code, it will everywhere — I don’t think this is something we are willing to risk breaking by adding first-class support for situational token passing. I think this also reveals some dangers with registering your flows per-user in CI. That said, the

Client

loads the auth token from the

prefect.context

which provides a context manager specifically for situations where the context should be temporarily modified so you can use that as a solution if you must. We have plans to work on service accounts in the near future and improving the cloud user/auth experience is on our roadmap so hopefully we can reduce the complexity of this for you!

Mitchell Bregman

10/21/2020, 7:08 PM

Awesome, thanks so much!!! I really appreciate the insight 🙂

3 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.