https://prefect.io logo
#prefect-community
Title
# prefect-community
x

Xavier Witdouck

06/19/2022, 4:45 PM
Hi all, is it possible to override a default parameter value when register a flow via the CLI? For example, our flows have an environment parameter (eg dev, qa, prod) and we don’t want to hard code that in the flow code. Ideally when we register the flow, we want to set the default value for that parameter. Thanks in advance.
a

Anna Geller

06/19/2022, 4:50 PM
The default parameter values get defined via code and you can then set custom values at runtime - the registration command is not used to define the workflow definition (such as Parameter tasks and their default values), it's only used to issue a command to send workflow metadata to the Prefect backend Check this for more details on parametrization in 1.0 https://www.prefect.io/guide/blog/how-to-make-your-data-pipelines-more-dynamic-using-parameters-in-prefect/
k

Kevin Kho

06/19/2022, 11:05 PM
I wanna add 2.0 definitely handles this a lot better
x

Xavier Witdouck

06/20/2022, 6:28 PM
Thanks. Unfortunately none of these approaches solve this rather basic problem. We need to specify the default parameters at registration, and it can’t be in the flow code itself. Is there a way at runtime to detect the labels of the agent that is running the flow? Since you can specify a label when you register a flow, we could use the label to infer parameters at runtime, if that makes sense?
a

Anna Geller

06/20/2022, 6:45 PM
Can you explain the problem you are trying to solve? Where are you in your Prefect adoption - have you used Prefect for some time already or are you rather new to Prefect?
x

Xavier Witdouck

06/20/2022, 6:55 PM
We are using Prefect in production, for about the past 4 weeks. We switched from using a proprietary orchestration engine. So our problem is we have 3 environments, DEV, QA, and PROD. We want to define flows generically, and them register them into each environment without making any code changes (ie changing default parameter values). At runtime, we need to be able to know which environment the flow is running against. We have agents called d3x_dev1, d3x_qa1, d3x_prod1 etc…so perhaps if we can detect the agent label at runtime. we can know which env the flow is running in. Does that make sense?
a

Anna Geller

06/20/2022, 7:25 PM
Yes, that definitely makes sense. Thanks so much for explaining your use case. Are you a Prefect Cloud customer? Or how are your different environments separated? If you are on Cloud, you could leverage the KV Store to accomplish that Otherwise, you could build a separate task that returns some default parameter value depending on the environment and passes that to the Parameter task We have a Discourse page about this problem: https://discourse.prefect.io/t/how-to-distinguish-between-various-environments-development-staging-production-in-prefect-flows/179 - one option might be to build some extra abstraction that sets different values based on the environments (such as set_run_config on Discourse)
I'm aware that all of those are a bit hacky. We're working on improving that UX in Prefect 2.0
4 Views