https://prefect.io
Join Slack
Hi, is there a way to access the dask's `client` object from Prefect's DaskExecutor? I need to be a...
# ask-community
s
Hi, is there a way to access the dask's 
client
object from Prefect's DaskExecutor? I need to be able to upload a python egg to the Dask cluster before starting executing Tasks (which I would achieve usually with 
client.upload_file)
Thank you in advance!
k
Hi @Sébastien Arnaud! I'm not sure 100% right now but this might help you?
And this
s
Thank you @Kevin Kho will take a look.
k
Just checking if this worked for you
s
Sorry @Kevin Kho I was out of the office today, I plan to test it tomorrow. Will let you know!
Looks like it could work @Kevin Kho if there was an easy way to upload a file (egg) to all the workers, or to create some kind of tasks to download the egg on each client first. before trying to trigger the egg install.
It looks like what I am looking for really is to be able to call this in Prefect: https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.UploadFile
k
Are you spinning up the cluster before running Prefect or as you run the Flow?
s
as the flow runs, but I could run it before hand (I use coiled for scale, local Dask cluster for test/development)
I would love to be able to control the cluster outside of Prefect so that I could easily scale up/down more easily and pass direct commands to the Dask client. Is there a way to do this?
k
I saw Chris answered your question. Looks like you have what you need?
s
Yes, this helped a lot. However, how to I get the dask client from the inside the plugin_handler function. Isn't the same question we were discussing earlier? 
def plugin_handler(f, old_state, new_state):
if new_state.is_running(): # flow run about to begin
# get dask client, register plugin if it doesn't exist
k
Not exactly sure but will this not work for you?
s
OK, then reusing worker_client() - will test it and see if it lets me register a plugin using that. Thanks again!
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by