s

    Stephen Lloyd

    5 months ago
    What is the standard way to set up prefect code to handle promotion through various environments? (dev -> stage -> prod). I’m using Prefect Cloud and would like to use the Secrets manager.
    Anna Geller

    Anna Geller

    5 months ago
    In Prefect 1.0, you would usually do this through labels. You can have an agent with the label
    "dev"
    . Then, you can change the label on the run configuration to "stage" to move it to this another environment (agent). And the easiest way to do it is to reregister your flow with an updated run configuration. Does this answer your question? How would you want to use Secrets manager?
    you could also have some custom logic to make it easier to set run config and storage based on the environment you want to deploy your flow to: example utility and usage example
    s

    Stephen Lloyd

    5 months ago
    Clarifying… It seems like you’re saying the preferred Prefect setup would be to: • run all three environments on a single AWS account, • but use different agents for each environment • re-register the flow when moving from one environment to another Is this close? I need to manage different AWS access keys to run prefect in different AWS accounts for each environment. I’d like to setup the flows to grab the correct keys depending on which environment the flow is running in.
    Anna Geller

    Anna Geller

    5 months ago
    Correct, a separate agent for each environment, and you will be able to match flows with agents via labels. To then promote a flow from dev to prod, you could reregister this flow with label dev, or you could have a separate flow (e.g. in project "dev") say FLOW_NAME_dev and register it with this separate agent label. Re AWS: you can have different credentials per AWS profile and switch between those
    s

    Stephen Lloyd

    5 months ago
    Interesting. We were planning to keep AWS access keys in Prefect Cloud Secrets and dynamically switch secret names to get the appropriate keys. We can use the same mechanic to implement something like your
    set_run_config()
    function. We use terraform to setup the Prefect agent on ECS, so I can set an environment variable on the agent, but I’m not sure how to get that to the container running the flow. Any thoughts on how I could do this?
    Anna Geller

    Anna Geller

    5 months ago
    If you are using ECS agent, you shouldn't need to use Prefect Secrets or manually set AWS credentials, instead you need to use IAM roles, let me send you some resources and examples
    s

    Stephen Lloyd

    5 months ago
    I really appreciate your help.