Hi there I'm wondering how I should store config-f...
# prefect-community
m
Hi there I'm wondering how I should store config-files in prefect 2. I saw the options with json-blocks, but I think it would be great to also have an yaml-block. Is there something like this? Maybe with even a bigger field or even a yaml editor helper. Background for this request is: we have a lot of configuration (connection details, mail server settings, special modes) and not everything should be possible to configure for every flow run as parameters. In prefect 1.0 we moved them to the config which was read by prefect.config (still possible to overwrite it with env-vars, even just from a flow run). As parameters we would like to use only such which are important for the daily execution. For all the password I will use the secrets. But I ask more about the normal configuration. Thanks for any hint!
m
Hey @Michael Hadorn I think you'd probably be able to define a custom block to achieve this or something similar, https://docs.prefect.io/concepts/blocks/#creating-new-block-types
m
@Mason Menges Thanks for you answer! I will try it. Btw: Do you have also an idea how to manage flow-runs for different environments (dev, tst, prod)? E.g. ETL on TST and PRD DB. The same flow (maybe different versions while developing), different deployments with different configuration. Is the best separation over different agents where I can specify some env vars for mark the environment, which I can use for building the correct names for block-configs and deployments names? Or do I find some collection of similar problems somewhere?
j
Hi Michael. You can have different deployments with different parameters and the same flows. That’s often a good way to solve the problem. Does that work for your use case?
m
Hi @Jeff Hale Thanks for your reply! Hm, this is like to many parameters. We have we have often to adjust, these will be added as params. But there are others which will not change (and if we can do without gui). So what would you recommend? Best Michael