Eddie Atkinson
05/06/2022, 2:48 AMFargateCluster
. It seems like the cluster is being assigned a public IP address. Ideally I wouldn’t want that to be the case as I don’t want people snooping on my cluster / submitting jobs. However, when I pass "fargate_private_ip": True
to cluster_kwargs
my cluster fails to start with the error: Cluster failed to start: Timed out trying to connect to <tcp://10.0.1.111:8786> after 30 s
That makes sense. Someone somewhere failed to connect to a local IP address, presumably from outside the subnet. What I don’t understand is how I can prevent people from arbitrarily accessing my cluster from the internet whilst allowing all the ‘right’ traffic throughKevin Kho
Anna Geller
What I don’t understand is how I can prevent people from arbitrarily accessing my cluster from the internet whilst allowing all the ‘right’ traffic throughOnly authorized IAM users/processes would be able to submit ECS tasks to your cluster - having access to your subnet wouldn't be enough to submit an ECS task. Then, only ECS tasks with a valid execution role ARN (one that has a trust policy with Action
"sts:AssumeRole"
) are able to start any container - without a valid role, even if a task would get submitted to ECS, no container within this ECS task could be started.
Here is how you can configure that with FargateCluster (full example here):
def fargate_cluster(
n_workers=2, image: str = "annageller/prefect-dask-cloudprovider:latest"
):
return FargateCluster(
n_workers=n_workers,
image=image,
execution_role_arn=f"arn:aws:iam::{AWS_ACCOUNT_ID}:role/prefectECSAgentTaskExecutionRole",
)
with Flow(
FLOW_NAME,
storage=STORAGE,
run_config=RUN_CONFIG,
executor=DaskExecutor(
cluster_class=fargate_cluster,
cluster_kwargs={
"image": "annageller/prefect-dask-cloudprovider:latest",
"n_workers": 2,
},
debug=True,
),
) as flow:
...
Eddie Atkinson
05/06/2022, 12:33 PMOnly authorized IAM users/processes would be able to submit ECS tasks to your cluster - having access to your subnet wouldn’t be enough to submit an ECS task.I am just testing this now by trying to connect to a running cluster with one set of authorised AWS creds and another set of unauthorised creds. Is there anywhere where I can read about this more / is there a reference to the Dask docs which I could include in my PR? I can imagine it will raise a few eyebrows if I submit a PR which looks like it exposes a cluster to the internet without justification
Eddie Atkinson
05/06/2022, 1:29 PMfrom dask_cloudprovider.aws import FargateCluster
kwargs = {
"image": "my-image-containing-dask",
"task_role_arn": "arn:aws:iam::xxxx",
"execution_role_arn": "arn:aws:iam::xxxx",
"n_workers": 10,
"region_name": "ap-southeast-2"
}
cluster = FargateCluster(**kwargs)
print(cluster.scheduler_address)
# use breakpoint so we can easily close cluster when we're done
breakpoint()
Once I have the scheduler’s address I switch my AWS profile to another AWS account without perms and run the flow below using Prefect to connect to the existing cluster:
import prefect
from prefect import task, Flow
from prefect.executors import LocalDaskExecutor, DaskExecutor
@task
def test_task():
print("running")
return 2 + 2
with Flow(
"test_flow",
) as flow:
test_task()
flow.executor = DaskExecutor(address="<ip-address>")
flow.run()
RuntimeWarning: coroutine 'rpc.close_rpc' was never awaited scheduler_comm.close_rpc()
Anna Geller
fargate_use_private_ip=True
but I can't help with the details about that - this is something you could ask here: https://dask.discourse.group/Eddie Atkinson
05/06/2022, 11:53 PMAnna Geller
Eddie Atkinson
05/07/2022, 12:11 AMsecurity_groups
and subnet_ids
in your cluster_kwargs