does anyone have good patterns for registering flows via ci/cd? ideally id like to register only the flows that changed automatically on a git push, and be able to use different run configs for each, while also handling env vars that may be different in prod/dev.
currently i have a script that loops through all my files in my flows directory and registers all flows using extract_flow_from_file. it runs on a push but the way i have it set up it:
• always registers all flows, regardless of whether they have changed or not
• has to pass the same env values to all flows, regardless of what the flow actually needs
• passes the same run config to all flows, not allowing me to customize memory requests per flow
any guidance on what to do here?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.