Hi, I’m using Prefect on ECS and have two separate...
# prefect-community
j
Hi, I’m using Prefect on ECS and have two separate clusters for test and prod. As suggested in one of the other threads here, I’m trying to use environment variables to specify the cluster name: However, when I run the flow in ECS it tells me the cluster is of type ‘None’ Here’s the run config I’m trying to use: Should using os.getenv work here?
a
j
Thanks Anna, yes, I’ve been looking at that. I can’t see how that would allow me to run in different ECS clusters using the same codebase though, unless you are using different AWS accounts and Prefect environments and pulling the secret AWS_ACCOUNT_ID will give a different value in each env?
I was actually hoping to implement something similar to Nate’s response to my question a few months back here: https://prefect-community.slack.com/archives/CL09KU1K7/p1651780170597659?thread_ts=1651773420.141339&cid=CL09KU1K7
I can’t understand why this works…
aws_account_id = Secret("AWS_ACCOUNT_ID").get()
…but this doesn’t seem to:
cluster = os.getenv('CLUSTER')
I also tried using secrets instead of environment variables (as shown in the example on discourse) and that doesn’t seem to work either
a
this repo has many ECSRun examples that might be helpful for you https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows
j
Thanks! I think I’m getting closer and think I’m doing it similarly to a couple of those examples there. With run_configs
local: bool = False
how is that set and what is it? A parameter? Is there a way of returning a different run config (even if we have a few different ones hard-coded) depending on the label? At the moment, I can only see how to do “not local” and “local”, if that makes sense
a
What is local?
j
I’m talking about this
Can set_run_config use something else apart from local to determine which label has been applied to the flowrun?
a
this is just a convenient hack to distinguish between env, you don't need to use it
j
thanks, but I’m actually trying to distinguish between two environments, staging and prod. Is there a similar hack to do that? The example only shows how to do it between local and prod
Another option would be to be able to read the labels (whether
staging
or
prod
) to be able to return the correct run_config, but from what I can see you can’t access those there
To elaborate, I can see that this works between “local” runs and production runs (non-local), but I can’t see how it would work for switching between staging and prod…