hi folks, how to set the docker storage configuration right? i have this example: ``` Docker( e...

Laura Vaida

02/26/2021, 4:36 PM

hi folks, how to set the docker storage configuration right? i have this example:

Copy code

Docker(

    env_vars={
        # append modules directory to PYTHONPATH
        "PYTHONPATH": "C://Users/laura.vaida.000/anaconda3/envs/prefect/Lib/site-packages"
    },
    stored_as_script=True
)

Laura Vaida

02/26/2021, 4:37 PM

error message is /opt/prefect/healthcheck.py151 UserWarning: Flow uses module which is not importable. Refer to documentation on how to import custom modules https://docs.prefect.io/api/latest/storage.html#docker flows = cloudpickle_deserialization_check(flow_file_paths) Traceback (most recent call last): File "/opt/prefect/healthcheck.py", line 151, in <module> flows = cloudpickle_deserialization_check(flow_file_paths) File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check flows.append(cloudpickle.loads(flow_bytes)) ModuleNotFoundError: No module named 'pandas'

J. Martins

02/26/2021, 4:40 PM

looks like you are missing ‘pandas’. You could simply provide Docker( …python_dependencies=[‘pandas’], ….)

J. Martins

02/26/2021, 4:41 PM

you can include any pip installable packages you need in that list. Hope this helps

Laura Vaida

02/26/2021, 4:46 PM

thanks tried that

Laura Vaida

02/26/2021, 4:47 PM

ValueError: A

path

must be provided to show where flow

.py

file is stored in the image.

Laura Vaida

02/26/2021, 4:47 PM

seems to be working but this error now appears

Laura Vaida

02/26/2021, 4:49 PM

• `path (str, optional)`: a direct path to the location of the flow file in the Docker image if

stored_as_script=True

Laura Vaida

02/26/2021, 4:49 PM

guess that arises because of my setting stored_as_Script = true, what path is meant by that?

J. Martins

02/26/2021, 4:50 PM

i guess so, if you skip setting stored_as_script, it should work

J. Martins

02/26/2021, 4:51 PM

otherwise you need to provide the path where the script is located

Zanie

02/26/2021, 4:53 PM

If you use

stored_as_script

you are responsible for copying the flow file into the Docker storage then providing a path to it. If you do not set that, then the flow pickled and copied into the Docker image for you

Laura Vaida

02/26/2021, 5:00 PM

if i use gcp should this be a alocal path?

J. Martins

02/26/2021, 5:04 PM

I guess you mean GCS?

Laura Vaida

02/26/2021, 5:04 PM

i use docker storage on google cloud platform

J. Martins

02/26/2021, 5:05 PM

if you use GSC and stored_as_script=True you will need to provide local_script_path=

Laura Vaida

02/26/2021, 5:07 PM

no i dont think

Laura Vaida

02/26/2021, 5:08 PM

Traceback (most recent call last): File "C:/Users/laura.vaida.000/Prefect/Billwerk/Billwerk_Contracts.py", line 175, in <module> flow.storage= Docker(registry_url="gcr.io/keller-steering-enabling/flows", image_name="billwerk-contracts",python_dependencies=["pandas"], File "C:\Users\laura.vaida.000\anaconda3\envs\prefect\lib\site-packages\prefect\storage\docker.py", line 222, in init super().__init__(stored_as_script=stored_as_script, **kwargs) TypeError: __init__() got an unexpected keyword argument 'local_script_path'

Laura Vaida

02/26/2021, 5:08 PM

Laura Vaida

02/26/2021, 5:08 PM

i use container registry

J. Martins

02/26/2021, 5:11 PM

If you want to store the flows in the container as a script check Michaels answer

J. Martins

02/26/2021, 5:12 PM

otherwise you can skip setting stored_as_script

Laura Vaida

02/26/2021, 5:12 PM

ah okey didn't get that at the first moment

Laura Vaida

02/26/2021, 5:12 PM

thanks! 🙂

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.