William Grim
02/23/2022, 6:41 PMDockerStorage
, if we can, so that registering flows brings along dependencies. We also want to use KubernetesRun
so that prefect can schedule jobs on k8s. For the latter, however, is KubernetesRun
still the right thing to do? Someone outside this community mentioned to us about dask for scheduling instead of the "kubernetes scheduler" (his words; I'm still new to this and figuring it out).
Basically, which way is the "right way"? Adding dask isn't something we're opposed to doing, but we have a lot of stuff going on and are aiming for the lowest effort path, haha. I promise we're not lazy, just overworked.Anna Geller
KubernetesRun
- here is an example. And you can use any executor with that as well as long as it's defined in your flowWilliam Grim
02/23/2022, 6:56 PMDockerStorage
and eventually move over to KubernetesRun
then. This is very helpful.Kevin Kho
William Grim
02/23/2022, 11:34 PMWilliam Grim
02/23/2022, 11:39 PMDockerStorage
(still using LocalRun
for the time being), and I have a Dockerfile
that's currently pretty empty, just to get one in place. It looks exactly like this:
FROM tiangolo/uvicorn-gunicorn-fastapi:python3.8-slim
Then I hacked together a DockerStorage
instantiation exactly like this (with the account id removed):
return storage.Docker(
registry_url="<http://accountid.dkr.ecr.us-west-1.amazonaws.com|accountid.dkr.ecr.us-west-1.amazonaws.com>",
image_name=flow.name,
image_tag="latest",
dockerfile="/workflows/Dockerfile", # this file exists in our docker container
)
But when I run this command from within the docker container where we have always registered flows, I see output like this:
Tenant id: 604a7da3-df0c-4639-aef8-58c228f30829
API cloud hook id: 01df50e6-02e3-41c9-9c6b-6e71eb1226d0
Connector project id: c684f02b-88d1-4cf5-af38-c563fcab9bb2
Collecting flows...
Processing 'flows/example_flow.py':
Building `Docker` storage...
Error building storage:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.8/http/client.py", line 1256, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1302, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1251, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1011, in _send_output
self.send(msg)
File "/usr/local/lib/python3.8/http/client.py", line 951, in send
self.connect()
File "/usr/local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory
And then several more lines, each with basically this exception:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/cli/build_register.py", line 463, in build_and_register
storage.build()
File "/usr/local/lib/python3.8/site-packages/prefect/storage/docker.py", line 308, in build
self._build_image(push=push)
File "/usr/local/lib/python3.8/site-packages/prefect/storage/docker.py", line 340, in _build_image
client = self._get_client()
File "/usr/local/lib/python3.8/site-packages/prefect/storage/docker.py", line 554, in _get_client
return docker.APIClient(
File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
self._version = self._retrieve_server_version()
File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file
or directory'))
William Grim
02/23/2022, 11:39 PMKevin Kho
docker run hello-world
on the CLI?Kevin Kho
William Grim
02/23/2022, 11:43 PMKevin Kho
-v /var/run/docker.sock:/var/run/docker.sock
attached to your Docker run commandWilliam Grim
02/23/2022, 11:47 PMWilliam Grim
02/24/2022, 1:10 AMWilliam Grim
02/24/2022, 1:12 AMDockerStorage
above, except now we set dockerfile=/register/Dockerfile
, and that directory just has these in it: Dockerfile, app/, workflows/
. the Dockerfile
just has a few lines:
FROM prefecthq/prefect
ENV PIP_NO_CACHE_DIR=1
RUN apt-get update
RUN apt-get upgrade -y
COPY requirements.txt ./
RUN pip install -r requirements.txt
COPY app /app
RUN pip install -e /app
COPY workflows /workflows
RUN pip install -e /workflows
ENV PYTHONPATH=/
but when i'm running our prefect register
command to register a flow, it starts up the docker storage build and then errors out with:
Processing '/workflows/flows/dbt_flow.py':
Building `Docker` storage...
2022-02-24 01:09:09+0000 - INFO - prefect.Docker - PID=178[MainProcess] - docker._build_image:358 - flow=[id=None; run_id=None; name=None
] - Building the flow's Docker storage...
Error building storage:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/docker/utils/build.py", line 97, in create_archive
t.addfile(i, f)
File "/usr/local/lib/python3.8/tarfile.py", line 1999, in addfile
copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
File "/usr/local/lib/python3.8/tarfile.py", line 255, in copyfileobj
raise exception("unexpected end of data")
OSError: unexpected end of data
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/cli/build_register.py", line 463, in build_and_register
storage.build()
File "/usr/local/lib/python3.8/site-packages/prefect/storage/docker.py", line 308, in build
self._build_image(push=push)
File "/usr/local/lib/python3.8/site-packages/prefect/storage/docker.py", line 364, in _build_image
output = client.build(
File "/usr/local/lib/python3.8/site-packages/docker/api/build.py", line 159, in build
context = utils.tar(
File "/usr/local/lib/python3.8/site-packages/docker/utils/build.py", line 29, in tar
return create_archive(
File "/usr/local/lib/python3.8/site-packages/docker/utils/build.py", line 99, in create_archive
raise IOError(
OSError: Can not read file in context: /proc/bus/pci/30ee:00/00.0
Registering 'dbt'... Error
================== 0 registered, 1 errored ==================
William Grim
02/24/2022, 1:13 AMWilliam Grim
02/24/2022, 1:18 AM/
and caused thisKevin Kho
William Grim
02/24/2022, 1:38 AMWilliam Grim
02/24/2022, 1:49 AMKevin Kho
William Grim
02/24/2022, 1:53 AMWilliam Grim
02/24/2022, 1:53 AMWilliam Grim
02/24/2022, 6:00 PMDockerStorage
which working directory it should use? I created a directory in /registrar
that contains Dockerfile, workflows/, app/
, and then I told DockerStorage
to use dockerfile=/registrar/Dockerfile
, but I don't think it knows to use that as the build context, because as soon as it reaches COPY workflows /workflows
, it comes out that it can't find that file in the build context.William Grim
02/24/2022, 6:02 PMKevin Kho
William Grim
02/24/2022, 6:19 PM