Stephen Lloyd
06/07/2022, 5:14 AMDaskExecutor
might be an option, but I don’t quite understand all the setup I would need to do.def set_run_config(flow_name: str) -> RunConfig:
aws_account_id = Secret('AWS_ACCOUNT_ID-' + RUN_ENV).get()
# aws_account_id = AWS_ACCOUNT_ID['dev']
return ECSRun(
cpu= 512,
memory= 1024,
run_task_kwargs=dict(cluster=prefect_cluster[RUN_ENV], launchType='FARGATE'),
execution_role_arn=f'arn:aws:iam::{aws_account_id}:role/prefectECSAgentTaskExecutionRole',
task_role_arn=f'arn:aws:iam::{aws_account_id}:role/prefectTaskRole',
image=f'{aws_account_id}.<http://dkr.ecr.us-east-1.amazonaws.com/{flow_name}:latest',\|dkr.ecr.us-east-1.amazonaws.com/{flow_name}:latest',\>
labels=[RUN_ENV],
)
Anna Geller
06/07/2022, 11:21 AMI’m running with 512 cpu and 1024 memoryin this setup, you don't have much capacity to parallelize things over, it's half of CPU core and 1 GB of memory - check this https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html
LocalDaskExecutor
in your flow running there and see if that helps - using DaskExecutor with the dask cloud provider FargateCluster brings a whole lot of complexity, so I would only use it as a last resort if increasing the capacity on your task definition didn't helpStephen Lloyd
06/07/2022, 11:25 AMbrings a whole lot of complexityIt seemed that way. Thanks!