dmo
10/16/2019, 1:49 PMdef get_answer_to_everything():
return 42
And then I define my flow in one_of_many_flows.py:
import my_utility
from prefect import Flow, task
@task
def run_imported_function():
return my_utility.get_answer_to_everything()
with Flow("Imported function") as flow1:
run_imported_function()
flow1.run() # 42
flow1.run(executor=DaskExecutor(address=IP)) # ModuleNotFoundError: No module named 'my_utility'
I know this is due to how the serialization works in Python. However, Dask provides a way to upload files to workers through Executor.upload_file:
https://stackoverflow.com/questions/39295200/can-i-use-functions-imported-from-py-files-in-dask-distributed
However, I have found no way to perform anything similar through the Prefect API. Am I looking in the wrong place or is this use case out of scope?Chris White
10/16/2019, 1:53 PMdmo
10/16/2019, 2:00 PMChris White
10/16/2019, 2:12 PMdmo
10/16/2019, 3:07 PMMaikel Penz
02/02/2020, 11:35 PMon_start
? I am using the DaskKubernetesEnvironment
and I get an error that my helper file cannot be imported when running itChris White
02/02/2020, 11:36 PMon_start
, but if you are using DaskKubernetesEnvironment
I’d instead recommend including the files that your Flow relies on within the Docker container storing your Flows.
Caveat: make sure that you also add the location of your helper files to your Docker image’s PATHMaikel Penz
02/02/2020, 11:39 PMfiles
setting on DockerStorage
to send my helpers along with the serialized version of my flow so they are there.. I believe the location could be missing from the PATH
then ...Chris White
02/02/2020, 11:39 PMDockerfile
so you can fully control how the image is builtMaikel Penz
02/02/2020, 11:40 PMPREFECT__CONTEXT__FLOW_FILE_PATH
right ?Chris White
02/02/2020, 11:46 PMCOPY /my/file.sh /my/file/in/docker/file.sh
ENV PATH="/my/file/in/docker:${PATH}"
or
COPY /my/file.py /my/file/in/docker/file.py
ENV PYTHON_PATH="/my/file/in/docker:${PYTHON_PATH}"
Maikel Penz
02/02/2020, 11:46 PMChris White
02/02/2020, 11:46 PM