Avi A08/23/2020, 11:13 AM
to run my flow on a local cluster. Where does prefect report the LocalCluster provisioning logs? I’m interested in what port the UI is served (port 8787 is already taken on the server)
Jenny09/08/2020, 2:19 PM
Avi A09/08/2020, 2:21 PM
but if for example if I’m running several flows on the same agent, with each of them spawning a local
, I have no idea what is the scheduler for the flow I’m currently running in. I hope this makes it a bit clearer.
Jenny09/08/2020, 2:23 PM
Avi A09/08/2020, 2:23 PM
Jim Crist-Harif09/08/2020, 2:48 PM
I actually also want to be able to connect to that scheduler from inside one of the tasks and I don’t know howI don't recommend doing this based on the address of the dask cluster, but rather use a
to get a client while inside a task running on a worker. Docs: https://distributed.dask.org/en/latest/api.html#distributed.worker_client
from distributed import worker_client @task def example(): with worker_client() as client: # do your dask stuff here
Avi A09/08/2020, 2:49 PM
so if I put Dask DataFrame operations inside the context in the code snippet you provided, will it run on the
client = Client(existing_cluster_addr) # this implicitly makes the client the default execution entry for dask operations ... dask_dataframe.operations.compute()
from the context?
Jim Crist-Harif09/08/2020, 3:02 PM
to specify which client to use.
Avi A09/08/2020, 3:03 PM
can only receive an address. Thanks a lot