Jimmy Le
02/08/2021, 5:28 PMTyler Wanner
02/08/2021, 5:34 PMFelix Vemmer
02/08/2021, 9:58 PMJimmy Le
02/10/2021, 3:48 PMsupervisord
to work on it. π€Billy McMonagle
02/10/2021, 3:51 PMsupervisord
? it's an option anyway.Jimmy Le
02/10/2021, 3:53 PMBilly McMonagle
02/10/2021, 3:55 PMsupervisord
for you. I believe it is correct to say that the task is part of the service (or the service runs the task)Jimmy Le
02/10/2021, 3:56 PMRUN
commands?Billy McMonagle
02/10/2021, 3:57 PMJimmy Le
02/10/2021, 4:00 PMsupervisord
where the flow was stored and which virtual environment it should use. It would then make sure all the agents were running.
But now that all the dependencies are nicely wrapped up in the ECS image, I don't currently see a reason to create so many labels.
Of course, I'm learning and would be open to any insights you've discovered.Billy McMonagle
02/10/2021, 4:01 PMAmanda Wee
02/10/2021, 5:18 PMENTRYPOINT
, then from that shell script I create projects, register flows, and finally start the local agent.
If my team decides to move ahead with running flows as Fargate tasks, what I'd try first is to modify the shell script to run a second agent, i.e., the ECS agent. Each agent will have its own label and flows will be labelled accordingly. This way, flows that are fast and don't consume much CPU/memory can continue to be handled by the local agent, while others will be handled by the ECS agent, and I can downsize the CPU/memory limits for this ECS service.
It won't quite fit the paradigm of one ECS task corresponding to one process (well it does, except that that process is a shell script spawning the prefect agents as subprocesses, and the local agent spawns a subprocess for each flow run), but it'll be less of a hassle than setting up another ECS service (as we do so through terraform assisted by our infrastructure team).
There is a difference in setup in that we're using S3 storage, so we'll need to create the Docker image for the ECS agent during our CI/CD process rather than as a result of using Docker storage.Jay Sundaram
03/11/2021, 8:24 PMbotocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: Task definition does not support launch_type FARGATE.
I've gone over the article several times to verify that I've set up properly- everything looks good.
Looking for troubleshooting tips from anyone. Thanks!Jimmy Le
03/11/2021, 8:26 PMJay Sundaram
03/11/2021, 8:27 PMaws ecs create-cluster --cluster-name my-fargate-cluster --capacity-providers FARGATE
Jimmy Le
03/11/2021, 8:32 PMJay Sundaram
03/11/2021, 8:33 PMJimmy Le
03/11/2021, 8:38 PMJay Sundaram
03/11/2021, 8:39 PMRUN_CONFIG = ECSRun(run_task_kwargs={'cluster': 'my-fargate-cluster'},
execution_role_arn='arn:aws:iam::<ACCOUNT>:role/<ROLE>,
labels=['ecs', 'dev'])
however i did add the following to the run_task_kwargs:
'launchType': 'FARGATE'
'requiresCompatibilities': ['FARGATE']