A question on the ECS Deployment block: • Is it possible to have an agent run in one AWS account (centrally), but have that agent schedule ECS Task in other accounts, so other ECS clusters? This is all assuming of course that the trust relationship between the accounts exists. • Why? Because running 20+ agents in different accounts turns the whole setup from being nicely flexible (“serverless”) into a cost nightmare. We set up multiple accounts and were hoping that the central account would succeed in scheduling the tasks in the other accounts, but got straight-up submission errors in the UI:

Submission failed. Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/prefect_aws/ecs.py", line 797, in _create_task_and_wait_for_start task = self._run_task(ecs_client, task_run) File "/usr/local/lib/python3.9/site-packages/prefect_aws/ecs.py", line 1458, in _run_task return ecs_client.run_task(**task_run)["tasks"][0] File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 508, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 915, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.ClusterNotFoundException: An error occurred (ClusterNotFoundException) when calling the RunTask operation: Cluster not found. The above exception was the direct cause of the following exception: RuntimeError: Failed to run ECS task, cluster 'prefect' not found. Confirm that the cluster is configured in your region.

Edit Bonus Question: How would you debug this kind of issue? The main AWS exceptions seems to be suppressed unfortunately…

There is an article I think on this issue - https://discourse.prefect.io/t/clusternotfoundexception-when-using-ecsagent/733/2 Generally account shouldn’t matter , but I would rule out Prefect here first - can you run

aws ecs list-clusters

and see the other accounts clusters. If you can’t, then I would expect it to be an AWS / permissions issue first

aws ecs list-clusters

and all of the AWS cli is on a per account basis, right? So it will only ever show the clusters from the active profile. Here’s the result from one of the accounts.

{
  "clusterArns": [
    "arn:aws:ecs:eu-central-1:123456789:cluster/prefect"
  ]
}

Can you confirm that an agent in cluster A can in principal schedule tasks on a different cluster? I briefly skimmed over the source code and it seemed unlikely, that that was possible…