https://prefect.io
Join Slack
Hi there, I am running into a weird issue where if I provide a `dockerfile` to the `prefect.environm...
# ask-community
w
Hi there, I am running into a weird issue where if I provide a 
dockerfile
to the 
prefect.environments.storage.Docker
class, I get an error that looks like this:
Copy code
shutil.Error "[Errno 63] File name too long: ['<Endless recursive path>']
a
@Wesam Manassra the environments are deprecated, so you should use a DockerRun run configuration instead
w
Thanks @Anna Geller. It looks like the old environment class used to manage python dependencies, but I don't see how that is done in the new 
DockerRun
class. Would appreciate any docs/pointers on that!
Btw, the class I mentioned earlier was not used for 
flow.environment
but rather 
flow.storage
That shouldn't be deprecated, right?
a
Sure! This is currently done on the Docker storage side. Here are some examples https://docs.prefect.io/orchestration/execution/storage_options.html#docker and API reference https://docs.prefect.io/api/latest/storage.html#docker e.g. you can pass a custom Dockerfile to the Docker storage
w
@Anna Geller I think there was a misunderstanding. I am already using the 
Docker
storage class, and this is the one I am having problems with.
a
Can you run prefect diagnostics and share your output? The correct import would be:
Copy code
from prefect.storage import Docker
can you share your full flow definition or at least the full Docker storage definition? It can be that the import path was just wrong. Can you try the same what you were doing but importing 
from prefect.storage import Docker
?
w
Our import is 
from prefect.environment.storage import Docker
We are using version 
0.13.16
It seems that the import you have was added in a later version? It fails for me
Flow config:
Copy code
flow.storage = Docker(
        python_dependencies=nonlocal_dependencies,
        registry_url="<http://gcr.io/big-notebook/flows|gcr.io/big-notebook/flows>",
        files=local_dependencies,
        dockerfile='Dockerfile',
        env_vars={
            # append modules directory to PYTHONPATH
            "PYTHONPATH": extend_python_path(local_dependencies.values())
        },
    )
    flow.environment = DaskKubernetesEnvironment()
    flow.state_handlers = [slack_notifier(only_states=[Failed])]
a
I see, that makes sense. If you want to upgrade, I would definitely encourage it. But regarding your error, this happens during registration right? it looks like you try to remove some file with a long file name?
w
Yes, the error happens during registration at the time of building the docker image
It is not that I am trying to remove the file, but rather the docker build fails with the error that some files with super long names exist.
Somehow it creates a local temporary directory for building (see this line) and that keeps recursively copying files, and causes this issue.
a
@Wesam Manassra can you share your Dockerfile? It’s hard to believe that a temporary directory alone would lead to shutil.Error, it must be something in the Dockerfile. Alternatively, you could build the image separately from the Docker build process that happens during registration. You would then use Docker storage only to add a flow object on top of a Docker image that already includes all your custom dependencies. This way you would have more control over the process and it can be perhaps easier to debug for you. Your flow storage would then look like:
Copy code
flow.storage = Docker(
    image_name="your_image_name", # name of the image you've already built
    image_tag="latest", # or a different tag
)
w
Copy code
FROM prefecthq/prefect:0.13.16-python3.8

apt-get install gcc
That's all of my dockerfile! Just trying to add some dependencies to the docker
a
I see. Yeah, building your own image will probably make it easier to debug the current shutil error issue. If you want an example, here is one Dockerfile and an example how it’s used in a project: https://github.com/anna-geller/packaging-prefect-flows/blob/master/Dockerfile
w
Sounds good, will give that a try instead. Thank you for your help, and for the example repo 🙂
🙌 1
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by