Jonathan Mathews
04/27/2022, 3:43 PMos.environ.get
? I have also tried EnvVarSecret
but that only seems to work within a flow?Kevin Kho
04/27/2022, 3:49 PMos.environ
. But I am a bit confused. Do you mean you want to perform this inside a task but EnvVarSecret is already a task so it can only be used in the Flow block?
If the environment variable sensitive info? Is it worth making a secret?Jonathan Mathews
04/27/2022, 3:51 PMDo you mean you want to perform this inside a task but EnvVarSecret is already a task so it can only be used in the Flow block?Yes, that’s right.
Kevin Kho
04/27/2022, 3:52 PMSecret.get()
inside tasks and you probably can do EnvVarSecret().run()
but it should just be equivalent to os.environ
Jonathan Mathews
04/27/2022, 3:52 PM)
set_run_config
and then defined as part of the flowKevin Kho
04/27/2022, 3:53 PMJonathan Mathews
04/27/2022, 3:54 PMKevin Kho
04/27/2022, 3:55 PMJonathan Mathews
04/27/2022, 3:56 PMKevin Kho
04/27/2022, 8:33 PMJonathan Mathews
04/27/2022, 8:36 PMtask_definition_path
or actually create the task definition and then reference it with task_definition_arn
Kevin Kho
04/27/2022, 8:42 PMimage
?Jonathan Mathews
04/27/2022, 8:48 PMenv
as below, but then they’re not secrets anymore as they appear in the logs!
ECSRun(
labels=["prod"],
image=f"{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.<http://amazonaws.com/prefect-dbt-snowflake:latest|amazonaws.com/prefect-dbt-snowflake:latest>",
env=dict(SNOWFLAKE_PASSWORD=SNOWFLAKE_PASSWORD,GIT_DEPLOY_TOKEN_PREFECT=GIT_DEPLOY_TOKEN_PREFECT),
execution_role_arn=f"arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectECSAgentTaskExecutionRole",
task_role_arn=f"arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectTaskRole",
Kevin Kho
04/27/2022, 8:50 PMcreate_flow_run
and notice you can pass a run_config
to this task here. What needs to happen is agent kicks of a “parent” flow with one task that kicks of a child flow with the configured RunConfiguration. This is the only avenue for run time dynamicismJonathan Mathews
04/27/2022, 9:02 PM