Thread
#prefect-community
    Jonathan Mathews

    Jonathan Mathews

    5 months ago
    Another question on ECS deployment.. what’s the best way to get environment variables set from my task definition? Can I use
    os.environ.get
    ? I have also tried
    EnvVarSecret
    but that only seems to work within a flow?
    Kevin Kho

    Kevin Kho

    5 months ago
    I think it would be
    os.environ
    . But I am a bit confused. Do you mean you want to perform this inside a task but EnvVarSecret is already a task so it can only be used in the Flow block? If the environment variable sensitive info? Is it worth making a secret?
    Jonathan Mathews

    Jonathan Mathews

    5 months ago
    Do you mean you want to perform this inside a task but EnvVarSecret is already a task so it can only be used in the Flow block?
    Yes, that’s right.
    Kevin Kho

    Kevin Kho

    5 months ago
    Yeah everything sounds right to me unless you pull it in the flow then pass it to the task. Just note you can also use
    Secret.get()
    inside tasks and you probably can do
    EnvVarSecret().run()
    but it should just be equivalent to
    os.environ
    Jonathan Mathews

    Jonathan Mathews

    5 months ago
    What I’m trying to do is populate the below dynamically: ECSRun( labels=[“prod”], image=f”{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.amazonaws.com/prefect-dbt-snowflake:latest”, execution_role_arn=f”arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectECSAgentTaskExecutionRole”, task_role_arn=f”arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectTaskRole”, run_task_kwargs=dict(cluster=“prefectECSCluster”, launchType=“FARGATE”,),
    )
    because is called within
    set_run_config
    and then defined as part of the flow
    Kevin Kho

    Kevin Kho

    5 months ago
    This can’t be populated dynamically during runtime because this is all metadata that’s registered during flow build time. These won’t be filled when the Flow starts, it’s filled even before the flow is retrieved
    What is the goal here? Are you trying to register across environments or something like that?
    Jonathan Mathews

    Jonathan Mathews

    5 months ago
    Ah ok, thanks. I’ll go back to hard coding it then
    Yes, I’m just writing some code that I can give to a client and it will work for them with minimal changes, but I can get them to hard code it, no problem
    Kevin Kho

    Kevin Kho

    5 months ago
    Ah ok was gonna say a pattern some people use is just having the flow definition as a function like you mentioned and then passing those in to fill
    Jonathan Mathews

    Jonathan Mathews

    5 months ago
    Ah I see, thanks for clarifying that
    I seem to be able to access secrets - SSM parameters - (as environment vars) in the agent task, but not in the runtask. I’ve been trying to figure out how to pass them down in the ECSRun method, following various approaches from the docs, but not having much luck. What might be the best way to do this? Any suggestions appreciated, as always!
    Kevin Kho

    Kevin Kho

    4 months ago
    What do you mean by agent task?
    Jonathan Mathews

    Jonathan Mathews

    4 months ago
    The task that is always running and that runs the Prefect agent
    (In ECS)
    I’ve been looking at this and wondering if I should either save another task definition in YAML in S3 and reference in ECSRun using
    task_definition_path
    or actually create the task definition and then reference it with
    task_definition_arn
    Kevin Kho

    Kevin Kho

    4 months ago
    Oh ECS Task lol. Just to be super clear, you have SSM parameters you want to pass as environment variables to ECSRun?
    Or are you trying to edit the ECSRun stuff like
    image
    ?
    Jonathan Mathews

    Jonathan Mathews

    4 months ago
    😄 Yes, that’s right, I’ve done it successfully using
    env
    as below, but then they’re not secrets anymore as they appear in the logs!
    ECSRun(
            labels=["prod"],
            image=f"{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.<http://amazonaws.com/prefect-dbt-snowflake:latest|amazonaws.com/prefect-dbt-snowflake:latest>",
    env=dict(SNOWFLAKE_PASSWORD=SNOWFLAKE_PASSWORD,GIT_DEPLOY_TOKEN_PREFECT=GIT_DEPLOY_TOKEN_PREFECT),
            execution_role_arn=f"arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectECSAgentTaskExecutionRole",
            task_role_arn=f"arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectTaskRole",
    Kevin Kho

    Kevin Kho

    4 months ago
    I think it’s not as simple as agent changing the run_config. You need a “parent” flow that calls
    create_flow_run
    and notice you can pass a
    run_config
    to this task here. What needs to happen is agent kicks of a “parent” flow with one task that kicks of a child flow with the configured RunConfiguration. This is the only avenue for run time dynamicism
    So I think the parent flow needs access to pull the secrets down and pass them on.
    Jonathan Mathews

    Jonathan Mathews

    4 months ago
    ok thanks, I’ll give that a try