https://prefect.io
Join Slack
Hi, I’m using Prefect on ECS and have two separate clusters for test and prod. As suggested in one o...
# ask-community
j
Hi, I’m using Prefect on ECS and have two separate clusters for test and prod. As suggested in one of the other threads here, I’m trying to use environment variables to specify the cluster name: However, when I run the flow in ECS it tells me the cluster is of type ‘None’ Here’s the run config I’m trying to use: Should using os.getenv work here?
a
j
Thanks Anna, yes, I’ve been looking at that. I can’t see how that would allow me to run in different ECS clusters using the same codebase though, unless you are using different AWS accounts and Prefect environments and pulling the secret AWS_ACCOUNT_ID will give a different value in each env?
I was actually hoping to implement something similar to Nate’s response to my question a few months back here: https://prefect-community.slack.com/archives/CL09KU1K7/p1651780170597659?thread_ts=1651773420.141339&cid=CL09KU1K7
I can’t understand why this works… 
aws_account_id = Secret("AWS_ACCOUNT_ID").get()
…but this doesn’t seem to: 
cluster = os.getenv('CLUSTER')
I also tried using secrets instead of environment variables (as shown in the example on discourse) and that doesn’t seem to work either
a
this repo has many ECSRun examples that might be helpful for you https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows
j
Thanks! I think I’m getting closer and think I’m doing it similarly to a couple of those examples there. With run_configs 
local: bool = False
how is that set and what is it? A parameter? Is there a way of returning a different run config (even if we have a few different ones hard-coded) depending on the label? At the moment, I can only see how to do “not local” and “local”, if that makes sense
a
What is local?
j
I’m talking about this
Can set_run_config use something else apart from local to determine which label has been applied to the flowrun?
a
this is just a convenient hack to distinguish between env, you don't need to use it
j
thanks, but I’m actually trying to distinguish between two environments, staging and prod. Is there a similar hack to do that? The example only shows how to do it between local and prod
Another option would be to be able to read the labels (whether 
staging
or 
prod
) to be able to return the correct run_config, but from what I can see you can’t access those there
To elaborate, I can see that this works between “local” runs and production runs (non-local), but I can’t see how it would work for switching between staging and prod…
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by