https://prefect.io logo
#prefect-community
Title
# prefect-community
m

Matt Alhonte

05/02/2022, 9:56 PM
Where in
ECSRun
would I put Dask security args? Like
tls_ca_file
and stuff?
k

Kevin Kho

05/02/2022, 10:03 PM
How would you do it without Prefect? Like this?
Copy code
from distributed import Client
from distributed.security import Security

sec = Security(tls_ca_file='cluster_ca.pem',
               tls_client_cert='cli_cert.pem',
               tls_client_key='cli_key.pem',
               require_encryption=True)

client = Client(..., security=sec)
m

Matt Alhonte

05/02/2022, 10:04 PM
Yeah exactly! And in
worker_extra_args
and
scheduler_extra_args
k

Kevin Kho

05/02/2022, 10:12 PM
Did you try:
Copy code
sec = Security(tls_ca_file='cluster_ca.pem',
               tls_client_cert='cli_cert.pem',
               tls_client_key='cli_key.pem',
               require_encryption=True)

executor = DaskExecutor(
    cluster_class="dask_cloudprovider.FargateCluster",
    cluster_kwargs={
        "image": "prefecthq/prefect:latest",
        "n_workers": 5,
        ...
        },
    client_kwargs={"security":sec}
)
?
m

Matt Alhonte

05/02/2022, 10:14 PM
Yep! It complained about passing data between the scheduler and the workers
So basically I'm trying to do Hyperparameter optimization - I wanna load the data in 1 task and then map the training task over the hyperparam combinations.
Lemme look through the CloudWatch logs for the exact error...
k

Kevin Kho

05/02/2022, 11:58 PM
^ DM’ed Matt. He will just read from the workers rather than passing the dataframe through the scheduler. Reading from workers directly is more efficient anyway
👍 1
8 Views