Where in

ECSRun

would I put Dask security args? Like

tls_ca_file

and stuff?

How would you do it without Prefect? Like this?

from distributed import Client
from distributed.security import Security

sec = Security(tls_ca_file='cluster_ca.pem',
               tls_client_cert='cli_cert.pem',
               tls_client_key='cli_key.pem',
               require_encryption=True)

client = Client(..., security=sec)

Yeah exactly! And in

worker_extra_args

and

scheduler_extra_args

Did you try:

sec = Security(tls_ca_file='cluster_ca.pem',
               tls_client_cert='cli_cert.pem',
               tls_client_key='cli_key.pem',
               require_encryption=True)

executor = DaskExecutor(
    cluster_class="dask_cloudprovider.FargateCluster",
    cluster_kwargs={
        "image": "prefecthq/prefect:latest",
        "n_workers": 5,
        ...
        },
    client_kwargs={"security":sec}
)

?

Yep! It complained about passing data between the scheduler and the workers

^ DM’ed Matt. He will just read from the workers rather than passing the dataframe through the scheduler. Reading from workers directly is more efficient anyway