https://prefect.io logo
Join Slack
Powered by
Hey folks :simple_smile: We are planning to upgrad...
# ask-community
a
Hey folks simple smile We are planning to upgrade our Prefect deployment from 0.13.13 to the latest version. We have 40+ flows, which are built using Prefect Core 0.13.13. Flows uses Docker storage and are stored on a private ECR repo. In productionthey run using the (deprecated) Prefect Fargate Agent, while for local development we use Prefect Docker Agent. Ideally, we’d like to upgrade Prefect Server and Prefect Agents to 0.15, and keep flows at Prefect Core 0.13.13 since we can’t upgrade them massively. I made some testing locally with Prefect Server and Prefect Docker Agent 0.15 and everything seems to work fine, but I hadn’t time to test extensively. Any suggestions on how to handle this upgrade process smoothly? 😊 Thanks!
k
See this about 0.15.0 at the moment for server. I think upgrading server and agents to 0.15, while keeping flows at 0.13.13 may break. Normally the minor version increment will contain breaking changes. If it works for you, that’s great! It does give weird errors though when it doesn’t work. I think you’d have to move from FargateAgent (maybe to ECSAgent?) when it gets deprecated though.
a
So maybe we better upgrade server and agent to a less recent version, say 0.14.x in order to minimize risks while keeping flows at 0.13.13?
k
That will be fixed next release and 0.15.0 might be worth the upgrade as it has some really convenient stuff. If you need to migrate now, then yeah I guess 0.14.21
a
I think we are going to upgrade first to 0.14.21, then upgrade flows and then upgrade to 0.15. Does this make sense?
k
Yep!
a
Thanks Kevin! 🙌
Hey @Kevin Kho simple smile Another question just came up to my mind 😅 With Fargate Agent we used to specify the 
containerDefinitions_environment
so that we can pass environment variables to the task that runs the flow. Is this still possible with the ECS Agent?
k
Let me check
a
👍
It was super helpful, because we declared the Agent task definition and the task environment using YAML in a CFN template
k
I believe using the 
env
of ECSRun will pass through
But you can also do it by supplying the task definition to ECSRun and it will be used when calling boto3's 
register_task_definition
.
a
mmmh…but in this way I need to know env var values during flow development/registration, while with 
containerDefinitions_environment
I can define env var at the agent level and then they are passed to the flow…
Basically I need to understand how to pass env var values from the Agent to the flow that Agent is going to start
Let me give an example of the CFN we use to deploy the Fargate Agent
Copy code
rAgentFargateTaskDefinition:
    Type: 'AWS::ECS::TaskDefinition'
    Metadata:
      cfn-lint:
        config:
          ignore_checks:
          - E1029
    DependsOn:
      - rFlowTaskRolePolicy
      - rAgentTaskRolePolicy
    Properties:
      TaskRoleArn: !Ref 'rAgentTaskRole'
      ExecutionRoleArn: !GetAtt 'rAgentExecutionRole.Arn'
      Family: !Sub '${pEnvironment}-prefect-fargate-agent'
      ContainerDefinitions:
        - Name: 'agent'
          Essential: true
          Command: ['prefect', 'agent', 'start', 'fargate', 'enable_task_revisions=true']
          Image: !Sub
            - 'prefecthq/prefect:${version}'
            - version: !If [ cIsPrefectServerVersionLatest, 'all_extras', !Sub 'all_extras-${pPrefectServerVersion}' ]
          MemoryReservation: 128
          Environment:
            - Name: PREFECT__CLOUD__API
              Value: !Sub '<masked value>'
            - Name: PREFECT__BACKEND
              Value: 'server'
            - Name: PREFECT__CLOUD__AGENT__LABELS
              Value: '["heavy"]'
            - Name: REGION_NAME
              Value: !Ref AWS::Region
            - Name: executionRoleArn
              Value: !GetAtt 'rFlowsExecutionRole.Arn'
            - Name: taskRoleArn
              Value: !GetAtt 'rFlowsTaskRole.Arn'
            - Name: networkMode
              Value: 'awsvpc'
            - Name: cluster
              Value: !Ref pEcsCluster
            - Name: networkConfiguration
              Value: !Sub
                - '{"awsvpcConfiguration": {"assignPublicIp": "DISABLED", "subnets": ["${subnets}"], "securityGroups": ["${pDWHClientSecurityGroup}"]}}'
                - subnets: !Join ['","', !Ref pFargateSubnetIds]
            - Name: cpu
              Value: '4096'
            - Name: memory
              Value: '8192'
            - Name: containerDefinitions_environment
              Value: !Sub
                - '[{
                      "name": "AWS_DEFAULT_REGION",
                      "value": "${AWS::Region}"
                    }, {
                      "name": "INTERNAL_SERVICES_CONNECTION_DATA_SECRET_ARN",
                      "value": "${rPrefectAgentInternalServicesConnectionDataSecret}"
                    }, {
                      "name": "DWH_JSON_CONNECTION_DATA_SECRET_ARN",
                      "value": "${rPrefectAgentDWHConnectionDataSecret}"
                    }, {
                      "name": "PLATFORM_DB_JSON_CONNECTION_DATA_SECRET_ARN",
                      "value": "${rPrefectAgentPlatformDBConnectionDataSecret}"
                    }, {
                      "name": "ENVIRONMENT",
                      "value": "${pEnvironment}"
                    }, {
                      "name": "DWH_QUERY_ROLE_ARN",
                      "value": "${pSafelakeDWHClusterRoleArn}"
                    }, {
                      "name": "EXTERNAL_SERVICES_SECRETS",
                      "value": "${rPrefectAgentExternalServicesSecret}"
                    }, {
                      "name": "DWH_PLATFORM_POSTGRES_USER_SECRET_ARN",
                      "value": "${pDWHPostgreSQLVersion11UserSecretArn}"
                    }, {
                      "name": "SAFELAKE_BUCKET_INGESTION_NAME",
                      "value": "${SafelakeBucketIngestionName}"
                    }, {
                      "name": "PREFECT__CONTEXT__SECRETS__SLACK_WEBHOOK_URL",
                      "value": "${pSlackWebhook}"
                    }, {
                      "name": "ROLLBAR_ENABLED",
                      "value": "true"
                    }, {
                      "name": "SLACK_ENABLED",
                      "value": "true"
                    }, {
                      "name": "PREFECT__SERVER__UI__ENDPOINT",
                      "value": "<masked_value>"
                    }
                   ]'
                - SafelakeBucketIngestionName: !Select [1, !Split [':::', !Ref pDatalakeBucketArn]]
            - Name: containerDefinitions_logConfiguration
              Value: !Sub '{ "logDriver": "awslogs", "options": { "awslogs-group": "${rFlowsLogsGroup}", "awslogs-region": "${AWS::Region}", "awslogs-stream-prefix": "prefect-flow"}}'
          LogConfiguration:
            LogDriver: 'awslogs'
            Options:
              awslogs-group: !Ref 'rAppLogsGroup'
              awslogs-region: !Ref 'AWS::Region'
              awslogs-stream-prefix: 'prefect-agent'
      Tags:
        - Key: 'Name'
          Value: !Sub '${pEnvironment}-prefect-fargate-agent'
        - Key: 'ApplicationVersion'
          Value: !Ref 'pApplicationVersion'
k
Values set using the 
--env
CLI flag on the agent are also passed through. Is that what you need?
a
Yeah I think it could work! Can we pass values using 
Environment
instead of the 
--env
CLI flag?
k
You mean setting it on the agent right? I’m honestly not sure, but I’d like to think you can
I’ll ask the team and get back
a
Yeah setting them in the Agent Task Defintion through the 
Environment
parameter. Otherwise we’ll have to provide all values in the 
command
parameter where we start the agent (passing values as you said using 
--env
CLI flag)
m
I believe you can use env_vars on ECSAgent, and here you can see the precedence of populating env vars
a
Hey @Mariia Kerimova simple smile Thanks! I understand that I can use 
--env
CLI flag to pass my env vars. I just need to understand if using the 
Environment
in the task definition of the agent would work as well simple smile
m
Just setting env var on agent task definition (without setting 
--env
CLI flag on agent or using 
ECSAgent(env_vars=..)
or 
ECSRun(env=..)
) will not pass the env var to the flow. I'll ask the team if we can have implicit env vars discovery
a
Sorry but it's a bit confusing 😅 What I'd like to do is to define env vars at the agent level, so that these variables are available to all flows. Ideally, i would do that by setting the variables in the Environment parameter or using --env CLI param in the Command parameter in the ECS task definition. Hopefully, these vars are passed along to the flow run without any special settings in ECSRun. Woukd this work?
Any suggestions? 😊
m
Hello @ale! Sorry for delay with reply! Yes, if you'll set 
--env
CLI param in the Command parameter in the ECS task definition, the environment variables will be passed to the flow 🙂
what I meant earlier is if you will just add env variables in Environment section of agent's task definition in CFN template without using 
--env
or 
ECSRun(env=..)
or 
ECSAgent(env_var)
, the variables will not be passed to the flow runs, but if you'll specify variables using 
--env
it should work 👍 I hope it makes sense, if not, let me know and I'll try to give you an example.
a
Hey @Mariia Kerimova 😊 An example on how to reference in 
--env
the values defined in the Environment would be super appreciated!
m
Sure, to pass with 
--env
will look like this in the console:
Or in your template set:
Copy code
Properties:
      TaskRoleArn: !Ref 'rAgentTaskRole'
      ExecutionRoleArn: !GetAtt 'rAgentExecutionRole.Arn'
      Family: !Sub '${pEnvironment}-prefect-fargate-agent'
      ContainerDefinitions:
        - Name: 'agent'
          Essential: true
          Command: ['prefect', 'agent', 'ecs', 'start', 'TEST_ECS_ENV=value_of_env_var']
then if you enabled CloudWatch, check the agent’s log, to make sure that env vars are set
and I tested it with flow:
Copy code
from prefect.storage import Docker
from prefect.executors import LocalExecutor
from prefect.run_configs import ECSRun
from prefect import task, Flow
import prefect
import os

STORAGE = Docker(
    registry_url="mariiaprefect",
    image_name="envvars-testflow",
)

RUN_CONFIG = ECSRun(run_task_kwargs={"cluster": "default-cluster"}, labels=["masha", "envtoken"])
EXECUTOR = LocalExecutor()

@task
def print_me():
    logger = prefect.context.get("logger")
    env_value = os.environ['TEST_ECS_ENV']
    <http://logger.info|logger.info>(f"Got the env var {env_value}")


with Flow(
    "Ale", storage=STORAGE, run_config=RUN_CONFIG
) as flow:
    print_me()
a
This is great! Thanks @Mariia Kerimova 😊 I assume we can also reference other CFN resources when assigning a value to TEST_ECS_ENV. For example, I should be able to set 
TEST_ECS_ENV=${pEnvironment}
where pEnvironment is a parameter defined in teh CFN template. Does this make sense?
m
I’ve never used CFN, but I think it should work!
a
Thanks 😊 Much appreciated!
👍 1
3 Views
Open in Slack
PreviousNext
Powered by