Riley Hun
01/11/2021, 6:43 PMKyle Moon-Wright
01/11/2021, 7:02 PMdask_gateway
to spin up a cluster manually or for advanced configurations - check out this advanced tutorial demonstrating both methods (Gateway method is under Advanced Dask Configuration
).Riley Hun
01/11/2021, 8:01 PMsecret_client = secretmanager.SecretManagerServiceClient()
response = secret_client.access_secret_version(
name="<path to secrets>")
secrets = json.loads(response.payload.data.decode('UTF-8'))
auth = BasicAuth(password=secrets['dask_gateway_secret'])
gateway = Gateway(
address=args.dask_proxy_address,
auth=auth
)
options = gateway.cluster_options()
options.image = args.dask_gateway_docker_image
cluster = gateway.new_cluster(options)
for flow in flows:
flow.storage = storage
flow.environment.executor = DaskExecutor(
address=args.dask_scheduler_address,
client_kwargs={'security': cluster.security}
)
path = storage.add_flow(flow)
print(f'storing flow {flow.name} at {path} at the image.')
Kyle Moon-Wright
01/11/2021, 8:24 PM