https://prefect.io logo
Title
k

Kevin Systrom

04/30/2020, 5:49 PM
Guys, is there a best practice for making sure the same code is available to workers on remote boxes? I've factored out a few functions into a utility python file that I import. I'm running dask-worker manually on a remote server and it complains (obviously) ModuleNotFoundError. I thought copying the code to remote and running worker from that directory might help but no dice
c

Chris White

04/30/2020, 5:53 PM
Hey @Kevin Systrom - I believe you’re one step away; if you add that directory to your
$PATH
prior to starting the dask-worker I’d expect you should be good, e.g.,
export PATH=/abs/path/to/directory:$PATH
dask-worker scheduler-address
k

Kevin Systrom

04/30/2020, 5:54 PM
good call. Always a manager never a sys admin 😉 thanks
c

Chris White

04/30/2020, 5:54 PM
haha anytime
j

Joe Schmid

04/30/2020, 6:08 PM
@Kevin Systrom maybe @Chris White already addressed everything you need, but we (long time Prefect users at a healthcare startup) have solved this with putting dependencies in docker images and having Dask workers use those. That might be overkill for your current setup, but holler if it's useful for me to tell you more.
:upvote: 1