Where in `ECSRun` would I put Dask security args? ...
# ask-community
m
Where in
ECSRun
would I put Dask security args? Like
tls_ca_file
and stuff?
k
How would you do it without Prefect? Like this?
Copy code
from distributed import Client
from distributed.security import Security

sec = Security(tls_ca_file='cluster_ca.pem',
               tls_client_cert='cli_cert.pem',
               tls_client_key='cli_key.pem',
               require_encryption=True)

client = Client(..., security=sec)
m
Yeah exactly! And in
worker_extra_args
and
scheduler_extra_args
k
Did you try:
Copy code
sec = Security(tls_ca_file='cluster_ca.pem',
               tls_client_cert='cli_cert.pem',
               tls_client_key='cli_key.pem',
               require_encryption=True)

executor = DaskExecutor(
    cluster_class="dask_cloudprovider.FargateCluster",
    cluster_kwargs={
        "image": "prefecthq/prefect:latest",
        "n_workers": 5,
        ...
        },
    client_kwargs={"security":sec}
)
?
m
Yep! It complained about passing data between the scheduler and the workers
So basically I'm trying to do Hyperparameter optimization - I wanna load the data in 1 task and then map the training task over the hyperparam combinations.
Lemme look through the CloudWatch logs for the exact error...
k
^ DM’ed Matt. He will just read from the workers rather than passing the dataframe through the scheduler. Reading from workers directly is more efficient anyway
👍 1