https://prefect.io logo
Title
j

Jonathan Mathews

04/27/2022, 3:43 PM
Another question on ECS deployment.. what’s the best way to get environment variables set from my task definition? Can I use
os.environ.get
? I have also tried
EnvVarSecret
but that only seems to work within a flow?
k

Kevin Kho

04/27/2022, 3:49 PM
I think it would be
os.environ
. But I am a bit confused. Do you mean you want to perform this inside a task but EnvVarSecret is already a task so it can only be used in the Flow block? If the environment variable sensitive info? Is it worth making a secret?
j

Jonathan Mathews

04/27/2022, 3:51 PM
Do you mean you want to perform this inside a task but EnvVarSecret is already a task so it can only be used in the Flow block?
Yes, that’s right.
k

Kevin Kho

04/27/2022, 3:52 PM
Yeah everything sounds right to me unless you pull it in the flow then pass it to the task. Just note you can also use
Secret.get()
inside tasks and you probably can do
EnvVarSecret().run()
but it should just be equivalent to
os.environ
j

Jonathan Mathews

04/27/2022, 3:52 PM
What I’m trying to do is populate the below dynamically: ECSRun( labels=[“prod”], image=f”{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.amazonaws.com/prefect-dbt-snowflake:latest”, execution_role_arn=f”arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectECSAgentTaskExecutionRole”, task_role_arn=f”arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectTaskRole”, run_task_kwargs=dict(cluster=“prefectECSCluster”, launchType=“FARGATE”,),
)
because is called within
set_run_config
and then defined as part of the flow
k

Kevin Kho

04/27/2022, 3:53 PM
This can’t be populated dynamically during runtime because this is all metadata that’s registered during flow build time. These won’t be filled when the Flow starts, it’s filled even before the flow is retrieved
What is the goal here? Are you trying to register across environments or something like that?
j

Jonathan Mathews

04/27/2022, 3:54 PM
Ah ok, thanks. I’ll go back to hard coding it then
Yes, I’m just writing some code that I can give to a client and it will work for them with minimal changes, but I can get them to hard code it, no problem
k

Kevin Kho

04/27/2022, 3:55 PM
Ah ok was gonna say a pattern some people use is just having the flow definition as a function like you mentioned and then passing those in to fill
j

Jonathan Mathews

04/27/2022, 3:56 PM
Ah I see, thanks for clarifying that
I seem to be able to access secrets - SSM parameters - (as environment vars) in the agent task, but not in the runtask. I’ve been trying to figure out how to pass them down in the ECSRun method, following various approaches from the docs, but not having much luck. What might be the best way to do this? Any suggestions appreciated, as always!
k

Kevin Kho

04/27/2022, 8:33 PM
What do you mean by agent task?
j

Jonathan Mathews

04/27/2022, 8:36 PM
The task that is always running and that runs the Prefect agent
(In ECS)
I’ve been looking at this and wondering if I should either save another task definition in YAML in S3 and reference in ECSRun using
task_definition_path
or actually create the task definition and then reference it with
task_definition_arn
k

Kevin Kho

04/27/2022, 8:42 PM
Oh ECS Task lol. Just to be super clear, you have SSM parameters you want to pass as environment variables to ECSRun?
Or are you trying to edit the ECSRun stuff like
image
?
j

Jonathan Mathews

04/27/2022, 8:48 PM
😄 Yes, that’s right, I’ve done it successfully using
env
as below, but then they’re not secrets anymore as they appear in the logs!
ECSRun(
        labels=["prod"],
        image=f"{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.<http://amazonaws.com/prefect-dbt-snowflake:latest|amazonaws.com/prefect-dbt-snowflake:latest>",
env=dict(SNOWFLAKE_PASSWORD=SNOWFLAKE_PASSWORD,GIT_DEPLOY_TOKEN_PREFECT=GIT_DEPLOY_TOKEN_PREFECT),
        execution_role_arn=f"arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectECSAgentTaskExecutionRole",
        task_role_arn=f"arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectTaskRole",
k

Kevin Kho

04/27/2022, 8:50 PM
I think it’s not as simple as agent changing the run_config. You need a “parent” flow that calls
create_flow_run
and notice you can pass a
run_config
to this task here. What needs to happen is agent kicks of a “parent” flow with one task that kicks of a child flow with the configured RunConfiguration. This is the only avenue for run time dynamicism
So I think the parent flow needs access to pull the secrets down and pass them on.
j

Jonathan Mathews

04/27/2022, 9:02 PM
ok thanks, I’ll give that a try