https://prefect.io logo
r

Ronald Sam

07/05/2023, 5:51 PM
Hi all, I’m trying to figure out what’s the best setup for us using prefect given our unique setup with gitlab. We have 3 environments that gitlab performs ci/cd on and a standalone server for developers (playground). The dev ops life cycle is below: 1. Develop and test in playground env 2. Push to gitlab under a feature branch 3. Deploy through pipeline in gitlab a. Stg b. QA c. Prod 4. Merge to main branch in gitlab PS. We’re not using containers. When we deploy to stg, qa, or prod using gitlab, it moves the python files/flows to those environments. If we’re using prefect and I’m deploying a flow from my local developer environment, because I don’t have a storage block it looks for the python file in the same folder path as my developer environment where I deployed the flow. Wondering if there’s a ingenious way for me to specify the default location depending on the environment the flow or work process is located that way the dev ops life cycle will be this instead: 1. Develop and test in playground env 2. Push to gitlab under a feature branch 3. Deploy through pipeline in gitlab a. Stg i. deploy flows that change with build steps for STG b. QA i. deploy flows that change with build steps for QA c. Prod i. deploy flows that change with build steps for Prod 4. Merge to main branch in gitlab Or if there’s a better/cleaner solution that I’m not thinking about please let me know.
r

redsquare

07/05/2023, 7:31 PM
use prefect enterprise with 3 workspaces, it's what I have in place
github actions with environment secrets
t

Taylor Curran

07/05/2023, 8:47 PM
Is there a reason why you can’t use a gitlab block to point to your flow code remotely? Setting up flow code storage that is local to a non-dev environment without containerization is not recommended. Remote code storage solutions from Prefect are designed to allow users to avoid needing to contend with such challenges.
You can have a gitlab pull step that pulls code from the feature branch for staging and QA. Then for prod this would be a pull step that pulls code from the main branch. This would mean 2 deployments for the same flow which is standard practice.
r

Ronald Sam

07/06/2023, 1:08 PM
I see what you mean, that would change how we're doing ci/cd with gitlab
and we have more than just python data pipelines. We use ssis, sql scripts and node js as well, as a result, we would need to have a separate deployment for python compared to the rest if we were to do it that way
we could, but would just prefer if we kept the deployment the same as what we have right now with gitlab
2 Views