https://prefect.io logo
r

Rolf Schick

12/15/2020, 3:21 PM
Hi. I’m new to Prefect and try to deploy a GreatExpectations environment into my docker/Kubernetes deployment. Unfortunately the only way I could get files into the container until now was by using the “files” parameter in the “Docker” storage class. To fill my container this way seems a bit odd to me. Is there another way I can just COPY my files into my container environment?
j

Jim Crist-Harif

12/15/2020, 3:24 PM
Hi Rolf, Ignoring GreatExpectations, if you want to get extra files into a Prefect-built docker image you have a few methods: • Pass in the
files
parameter to the
Docker
class • Pass a custom
dockerfile
that manually adds your files to the
Docker
class • Pre-build a new base image that already has your files added, then pass in that image as
base_image
to
Docker
.
r

Rolf Schick

12/15/2020, 3:32 PM
Thanks for the answer @Jim Crist-Harif. The “manual passing of a docker file” was the feature I was looking towards. I tried now relativ and absolute paths with the docker “COPY” command, but unfortunately Prefect still adds the prefix “/var/lib/docker/tmp/docker-builder994084773” before the path, resulting in an Error.
j

Jim Crist-Harif

12/15/2020, 3:40 PM
I tried now relativ and absolute paths with the docker “COPY” command, but unfortunately Prefect still adds the prefix “/var/lib/docker/tmp/docker-builder994084773” before the path, resulting in an Error.
Sorry, are you saying this happened with a custom dockerfile? Or using the
files
arg to
Docker
?
r

Rolf Schick

12/15/2020, 3:47 PM
until now I’ve done both. using the “files” arg and the “Docker” one. • “Docker” the prerequisites of “pyodbc” • “files” for the manual copying.
I was hoping to replace the “files” arg and rather use just the “docker” arg. Unfortunately I can’t use the COPY command from docker in the dockerfile.
j

Jim Crist-Harif

12/15/2020, 3:53 PM
Sorry, I'm unable to replicate, can you provide an example (something simple) that doesn't work the way you expect?
r

Rolf Schick

12/15/2020, 3:59 PM
I was looking to just copy a folder with all its content with the dockerfile. unfortunately that is not working due to a prefix which is added.
j

Jim Crist-Harif

12/15/2020, 4:25 PM
Hmmm, I'm unable to replicate.
Copy code
$ tree
.
├── Dockerfile
├── flow.py
└── samplefolder
    └── testfile

1 directory, 3 files

$ cat Dockerfile
FROM prefecthq/prefect:latest-python3.8

COPY samplefolder/. /opt/app/great_expectations/

$ cat flow.py
from prefect import Flow, task
from prefect.environments.storage import Docker

@task
def hello():
    print("hello")

with Flow("example") as flow:
    hello()

flow.storage = Docker(dockerfile="Dockerfile")

flow.register("testing")

$ python flow.py
...
Successfully built fcc4916c9281
Successfully tagged example:2020-12-15t16-23-44-416437-00-00
Flow URL: <https://cloud.prefect.io/jim-prefectio/flow/1debc0b1-aa7b-42b1-b6df-97051786279c>
 └── ID: 38ad0ce0-f222-432e-860e-c2fec8ac21eb
 └── Project: testing
 └── Labels: []
👍 1
Does the above work for you? If you're using a dockerfile, it should use the current working directory (the one you call
python flow.py
from) as the build context, rather than creating a temporary directory.
👍 1
r

Rolf Schick

12/15/2020, 4:29 PM
Thanks @Jim Crist-Harif I’ll debug my way from here.