Another question on ECS deployment.. what’s the best way to get environment variables set from my task definition? Can I use

os.environ.get

? I have also tried

EnvVarSecret

but that only seems to work within a flow?

I think it would be

os.environ

. But I am a bit confused. Do you mean you want to perform this inside a task but EnvVarSecret is already a task so it can only be used in the Flow block? If the environment variable sensitive info? Is it worth making a secret?

Do you mean you want to perform this inside a task but EnvVarSecret is already a task so it can only be used in the Flow block?

Yes, that’s right.

Yeah everything sounds right to me unless you pull it in the flow then pass it to the task. Just note you can also use

Secret.get()

inside tasks and you probably can do

EnvVarSecret().run()

but it should just be equivalent to

os.environ

What I’m trying to do is populate the below dynamically: ECSRun( labels=[“prod”], image=f”{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.amazonaws.com/prefect-dbt-snowflake:latest”, execution_role_arn=f”arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectECSAgentTaskExecutionRole”, task_role_arn=f”arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectTaskRole”, run_task_kwargs=dict(cluster=“prefectECSCluster”, launchType=“FARGATE”,),

)

This can’t be populated dynamically during runtime because this is all metadata that’s registered during flow build time. These won’t be filled when the Flow starts, it’s filled even before the flow is retrieved

Ah ok, thanks. I’ll go back to hard coding it then

Ah ok was gonna say a pattern some people use is just having the flow definition as a function like you mentioned and then passing those in to fill

Ah I see, thanks for clarifying that

What do you mean by agent task?

The task that is always running and that runs the Prefect agent

Oh ECS Task lol. Just to be super clear, you have SSM parameters you want to pass as environment variables to ECSRun?

😄 Yes, that’s right, I’ve done it successfully using

env

as below, but then they’re not secrets anymore as they appear in the logs!

ECSRun(
        labels=["prod"],
        image=f"{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.<http://amazonaws.com/prefect-dbt-snowflake:latest|amazonaws.com/prefect-dbt-snowflake:latest>",
env=dict(SNOWFLAKE_PASSWORD=SNOWFLAKE_PASSWORD,GIT_DEPLOY_TOKEN_PREFECT=GIT_DEPLOY_TOKEN_PREFECT),
        execution_role_arn=f"arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectECSAgentTaskExecutionRole",
        task_role_arn=f"arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectTaskRole",

I think it’s not as simple as agent changing the run_config. You need a “parent” flow that calls

create_flow_run

and notice you can pass a

run_config

to this task here. What needs to happen is agent kicks of a “parent” flow with one task that kicks of a child flow with the configured RunConfiguration. This is the only avenue for run time dynamicism

ok thanks, I’ll give that a try