Riley Hun
01/15/2021, 6:13 PMZanie
Riley Hun
01/15/2021, 6:28 PMauth = BasicAuth(password=secrets['dask_gateway_secret'])
gateway = Gateway(
address=args.dask_proxy_address,
auth=auth
)
options = gateway.cluster_options()
options.image = args.dask_gateway_docker_image
cluster = gateway.new_cluster(options)
cluster.scale(5)
for flow in flows:
flow.storage = storage
flow.executor = DaskExecutor(
address=cluster.scheduler_address,
client_kwargs={'security': cluster.security}
)
path = storage.add_flow(flow)
print(f'storing flow {flow.name} at {path} at the image.')
storage = storage.build()
for flow in flows:
flow.register(project_name=args.project, build=False)
Zanie
Riley Hun
01/15/2021, 7:06 PMcluster_class
argument. That's very cool functionality! ThanksZanie