Avi A
08/23/2020, 11:13 AMLocalEnvironment
/ DaskCluster
to run my flow on a local cluster. Where does prefect report the LocalCluster provisioning logs? I’m interested in what port the UI is served (port 8787 is already taken on the server)Jenny
09/08/2020, 2:19 PMAvi A
09/08/2020, 2:21 PMlocalhost:8786
but if for example if I’m running several flows on the same agent, with each of them spawning a local DaskCluster
, I have no idea what is the scheduler for the flow I’m currently running in. I hope this makes it a bit clearer.Jenny
09/08/2020, 2:23 PMAvi A
09/08/2020, 2:23 PMJim Crist-Harif
09/08/2020, 2:48 PMI actually also want to be able to connect to that scheduler from inside one of the tasks and I don’t know howI don't recommend doing this based on the address of the dask cluster, but rather use a
worker_client
to get a client while inside a task running on a worker. Docs: https://distributed.dask.org/en/latest/api.html#distributed.worker_client
from distributed import worker_client
@task
def example():
with worker_client() as client:
# do your dask stuff here
Avi A
09/08/2020, 2:49 PMclient = Client(existing_cluster_addr) # this implicitly makes the client the default execution entry for dask operations
...
dask_dataframe.operations.compute()
so if I put Dask DataFrame operations inside the context in the code snippet you provided, will it run on the client
from the context?Jim Crist-Harif
09/08/2020, 3:02 PMscheduler=client
to .compute
to specify which client to use.Avi A
09/08/2020, 3:03 PMscheduler
can only receive an address. Thanks a lot