https://prefect.io logo
Title
b

Braun Reyes

03/20/2020, 3:40 PM
Now that we have the concept of multiple schedules per flow...has there been any thought to concept of environments for CICD? Like deploy a flow then promote that version to dev -> uat -> prod? We kind of do that now but with different projects and actually registering the flow in different projects. Down side is that it’s 2 flows against your allotment. Would just have to think about ‘config’ per environment. Dbt Cloud does this. Also kind of inline with k8s namespaces and perhaps AWS elastic beanstalk
j

josh

03/20/2020, 5:38 PM
We’ve thrown around the idea of specifying environment specific configuration at runtime which poses some challenges but seems similar nonetheless. Jw, by promote dev -> … -> prod are you currently deploying a new version to a new project or are you changing the project of the flow to move it through?
b

Braun Reyes

03/20/2020, 6:55 PM
we have a <team-name>-uat and <team-name>-prod project...when we merge to master flows get registered to the uat project with proper docker image:tag assigned....when we cut github release we register to the prod project with proper image:tag assigned
perhaps this idea of context or using external kwargs and integrating with k8s configMaps or AWS parameter store/secrets manager to store the env specific configurations
i think it would probably be best just to simply tags flow_ids then build in a mechanism in the environments/agents to apply a configuration via tag. Then build it into the schedules to also tie into these tags...like a schedule only triggers flows tagged at prod/latest..something like that. Would need to also allow people to be able to trigger new versions for testing while also keeping the current prod version running and the schedule
c

Chris White

03/20/2020, 7:40 PM
Not sure if this helps with your workflow but FYI you can update a Flow’s project without re-registering it via the
update_flow_project
GraphQL mutation
b

Braun Reyes

03/20/2020, 7:45 PM
I guess I was just curious about supporting this workflow with in the context of a single flow, same way we support multiple schedules as way of running the same flows differently for each schedule
a

Adam Roderick

03/23/2020, 1:53 PM
We follow a similar process, but we deploy our flows with different names (e.g. *-prod) rather than putting them in different projects. Similar issues that Braun is facing
a

Alan Featherston

03/24/2020, 10:58 PM
For what is worth, we are also facing the same issue. Our ci/cd pipeline pushes all flows to a "test" project during development, and then to the right project when the merge request is approved and merge in master. A native solution would be much appreciated.
c

Chris White

03/24/2020, 11:01 PM
This is probably a non-standard place to take a poll, but if you could change your Flow’s labels natively in the UI / through GraphQL, would that solve the promotion problem for everyone?
@Alan Featherston / @Braun Reyes / @Adam Roderick
a

Alan Featherston

03/25/2020, 12:00 AM
First I thought, yeah that would be amazing. But on a second though and I might be misunderstanding how it would work but a Flow goes through several developement cycles, meaning that while a Flow is in dev/testing we still need to be able to run the "prod" version of it. Is it possible to have different labels for different versions of the Flow? Initially I had thought that a good solution would be to have a concept of "stages" and labels could be assigned to these stages, I hope that makes sense. I'm keen to hear what other had thought about possible solutions.
a

Adam Roderick

03/25/2020, 12:13 PM
Possibly. We deploy our flows with some environment variables that are environment-specific. Somehow we would have to inspect the flow name at runtime to determine what the environment-specific config would be
b

Braun Reyes

03/25/2020, 2:22 PM
so that way that we handle envars is that we build the image for each environments and put that environments envars in the docker image. dev gets dev envars and/or dev.toml, etc we also disable schedule on all non-prod flows The things that need to happen for environment specific workflow would be: • register a flow to a project with some sort of alias like 'dev', 'staging', 'uat', 'prod', etc. It would always be one version per alias. Obviously you should be able to manually run any of these aliased version for testing. Then finally be able to change the alias to flow version relationship via the API. • if a schedule is defined on the flow, have it attributed to these aliases to that you do something like, only schedule production. • in regards to environment configurations, the use of
config.toml
and native cloud/k8s components could work. for k8s, you can have an alias config map applied that is either envars or a config.toml file mounted as volume. for something like fargate/aws you could pull down a config.toml from s3 before running a container in an entrypoint script. doing the first 2 would go a long way and then last bullet could be handled in the CICD process of the user