Hi, is there a way to access the dask's `client` o...
# ask-communitys
Sébastien Arnaud
05/13/2021, 3:40 AMHi, is there a way to access the dask's
clientclient.upload_file)k
Kevin Kho
05/13/2021, 3:47 AM
Hi @Sébastien Arnaud! I'm not sure 100% right now but this might help you?
Kevin Kho
05/13/2021, 3:48 AM
And this
s
Sébastien Arnaud
05/13/2021, 3:50 AMThank you @Kevin Kho will take a look.
k
Kevin Kho
05/13/2021, 2:51 PM
Just checking if this worked for you
s
Sébastien Arnaud
05/14/2021, 4:14 AMSorry @Kevin Kho I was out of the office today, I plan to test it tomorrow. Will let you know!
Sébastien Arnaud
05/20/2021, 12:51 AMLooks like it could work @Kevin Kho if there was an easy way to upload a file (egg) to all the workers, or to create some kind of tasks to download the egg on each client first. before trying to trigger the egg install.
Sébastien Arnaud
05/20/2021, 12:57 AMIt looks like what I am looking for really is to be able to call this in Prefect: https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.UploadFile
k
Kevin Kho
05/20/2021, 2:28 AM
Are you spinning up the cluster before running Prefect or as you run the Flow?
s
Sébastien Arnaud
05/20/2021, 3:01 AMas the flow runs, but I could run it before hand (I use coiled for scale, local Dask cluster for test/development)
Sébastien Arnaud
05/20/2021, 3:02 AMI would love to be able to control the cluster outside of Prefect so that I could easily scale up/down more easily and pass direct commands to the Dask client. Is there a way to do this?
k
Kevin Kho
05/20/2021, 12:51 PM
I saw Chris answered your question. Looks like you have what you need?
s
Sébastien Arnaud
05/20/2021, 1:59 PMYes, this helped a lot. However, how to I get the dask client from the inside the plugin_handler function. Isn't the same question we were discussing earlier?
def plugin_handler(f, old_state, new_state):if new_state.is_running(): # flow run about to begin# get dask client, register plugin if it doesn't existk
Kevin Kho
05/20/2021, 2:00 PM
Not exactly sure but will this not work for you?
s
Sébastien Arnaud
05/20/2021, 4:23 PMOK, then reusing worker_client() - will test it and see if it lets me register a plugin using that. Thanks again!