Mohammed Siddiqui
01/30/2025, 8:27 AMdef generate_default_ecs_container_args(log_stream_prefix: str,
env=None
) -> dict:
if env is None:
env = {}
new_env = dict()
new_env.update(default_env)
new_env.update(env)
return {
'auto_deregister_task_definition': True,
'env': new_env,
'family': f"prefect-prod-mumbai-flow-run-{log_stream_prefix}",
# 'task_definition_arn': ECS_TASK_DEFINITION_ARN,
'execution_role_arn': ECS_EXECUTION_ROLE_ARN,
'task_role_arn': ECS_TASK_ROLE_ARN,
'network_configuration': {
'awsvpcConfiguration': {
'subnets': [ECS_VPC_SUBNET],
'securityGroups': [ECS_VPC_SECURITY_GROUP],
'assignPublicIp': 'DISABLED'
}
},
'configure_cloudwatch_logs': True,
'cloudwatch_logs_options': {
'awslogs-create-group': 'true',
'awslogs-region': AWS_REGION,
'awslogs-group': ECS_CLOUDWATCH_LOG_GROUP,
'awslogs-stream-prefix': log_stream_prefix
},
'stream_output': True,
'launch_type': 'FARGATE',
'vpc_id': ECS_VPC_ID,
'cluster': ECS_CLUSTER,
'task_start_timeout_seconds': 180,
'labels': {
'Team': ECS_CONTAINER_TEAM_TAG,
'rewaa_env': ECS_CONTAINER_REWAA_ENV_TAG,
'rewaa_team': ECS_CONTAINER_REWAA_TEAM_TAG
}
}
def get_default_deployment_args() -> dict:
return {
'work_pool_name': 'default',
'work_queue_name': 'default',
'build': False,
'push': False
}
default_build_args = {
'name': 'regular',
'image': CUSTOM_PREFECT_IMAGE,
**get_default_deployment_args()
}
some_flow.deploy(
job_variables={
'cpu': 1 * 1024,
'memory': 2 * 1024,
**generate_default_ecs_container_args(log_stream_prefix='ecs-some-flow', env=env)
},
**default_build_args,
I know we can reference a Task definition ARN and specify it there, but I don't want to create a custom script which would be responsible for creating custom task definitions for every flow (we have 20+ flows currently) and then referencing them in each flow separately.
This has happened because of a behavior change I have noticed upgrading from Prefect 2 to 3. In Prefect 2, if you provided a task_definition_arn, I think it overrides properties on it based on what custom properties you set during the deploy phase, but in Prefect 3 if you pass the task_definition_arn, it completely ignores any other job variables you pass to the ECS Worker.
Any suggestion or help would be appreciated!Jake Kaplan
01/30/2025, 3:28 PMJake Kaplan
01/30/2025, 3:29 PMMohammed Siddiqui
01/30/2025, 4:03 PMMohammed Siddiqui
01/30/2025, 4:03 PMJake Kaplan
01/30/2025, 4:42 PM