Jonathan Mathews
04/27/2022, 3:43 PMos.environ.get
? I have also tried EnvVarSecret
but that only seems to work within a flow?Kevin Kho
os.environ
. But I am a bit confused. Do you mean you want to perform this inside a task but EnvVarSecret is already a task so it can only be used in the Flow block?
If the environment variable sensitive info? Is it worth making a secret?Jonathan Mathews
04/27/2022, 3:51 PMDo you mean you want to perform this inside a task but EnvVarSecret is already a task so it can only be used in the Flow block?Yes, that’s right.
Kevin Kho
Secret.get()
inside tasks and you probably can do EnvVarSecret().run()
but it should just be equivalent to os.environ
Jonathan Mathews
04/27/2022, 3:52 PM)
Jonathan Mathews
04/27/2022, 3:53 PMset_run_config
and then defined as part of the flowKevin Kho
Kevin Kho
Jonathan Mathews
04/27/2022, 3:54 PMJonathan Mathews
04/27/2022, 3:54 PMKevin Kho
Jonathan Mathews
04/27/2022, 3:56 PMJonathan Mathews
04/27/2022, 8:28 PMKevin Kho
Jonathan Mathews
04/27/2022, 8:36 PMJonathan Mathews
04/27/2022, 8:36 PMJonathan Mathews
04/27/2022, 8:37 PMtask_definition_path
or actually create the task definition and then reference it with task_definition_arn
Kevin Kho
Kevin Kho
image
?Jonathan Mathews
04/27/2022, 8:48 PMenv
as below, but then they’re not secrets anymore as they appear in the logs!
ECSRun(
labels=["prod"],
image=f"{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.<http://amazonaws.com/prefect-dbt-snowflake:latest|amazonaws.com/prefect-dbt-snowflake:latest>",
env=dict(SNOWFLAKE_PASSWORD=SNOWFLAKE_PASSWORD,GIT_DEPLOY_TOKEN_PREFECT=GIT_DEPLOY_TOKEN_PREFECT),
execution_role_arn=f"arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectECSAgentTaskExecutionRole",
task_role_arn=f"arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectTaskRole",
Kevin Kho
create_flow_run
and notice you can pass a run_config
to this task here. What needs to happen is agent kicks of a “parent” flow with one task that kicks of a child flow with the configured RunConfiguration. This is the only avenue for run time dynamicismKevin Kho
Jonathan Mathews
04/27/2022, 9:02 PM