Vikram Iyer
07/01/2020, 7:02 PMmyapp
and another container which is actually the prefect agent
that will be running the flows. I figured out the agent
needs to have the code that is there in myapp
. So, I set up the flow like below,
flow.storage = Docker(dockerfile='path_of_Dockerfile_inside_myapp') #this will ensure that Dockerfile I used to build myapp will be used to build agent
flow.storage.build() #this step fails with the eroor: FileNotFound. I looked at the prefect code in the github repo and this should work, but fails
flow.register(labels=['some_label'])
Is there a gap in my understanding of how the Docker storage works?josh
07/01/2020, 7:06 PMdockerfile=…
Also as a side note, you might not need to build the storage directly because flow.register
has a build
kwarg which defaults to True
Vikram Iyer
07/01/2020, 7:11 PM/app/somedir/code.py
and The place where the Dockerfile is, is /app/Dockerfile
. Ideally the parent dir so should be accessible. What do you think?josh
07/01/2020, 7:12 PM.build
?Vikram Iyer
07/01/2020, 7:16 PMmyapp | /usr/local/lib/python3.7/site-packages/prefect/client/client.py:619: UserWarning: No result handler was specified on your Flow. Cloud features such as input caching and resuming task runs from failure may not work properly.
myapp | "No result handler was specified on your Flow. Cloud features such as "
myapp | /usr/local/lib/python3.7/site-packages/prefect/core/flow.py:1371: UserWarning: A flow with the same name is already contained in storage; if you changed your Flow since the last build, you might experience unexpected issues and should re-create your storage object.
myapp | "A flow with the same name is already contained in storage; if you "
myapp | [2020-07-02 00:43:18 +0530] [50] [ERROR] Error handling request /api/v1/metric/add/
myapp | Traceback (most recent call last):
myapp | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
myapp | chunked=chunked,
myapp | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request
myapp | conn.request(method, url, **httplib_request_kw)
myapp | File "/usr/local/lib/python3.7/http/client.py", line 1252, in request
myapp | self._send_request(method, url, body, headers, encode_chunked)
myapp | File "/usr/local/lib/python3.7/http/client.py", line 1298, in _send_request
myapp | self.endheaders(body, encode_chunked=encode_chunked)
myapp | File "/usr/local/lib/python3.7/http/client.py", line 1247, in endheaders
myapp | self._send_output(message_body, encode_chunked=encode_chunked)
myapp | File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output
myapp | self.send(msg)
myapp | File "/usr/local/lib/python3.7/http/client.py", line 966, in send
myapp | self.connect()
myapp | File "/usr/local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 43, in connect
myapp | sock.connect(self.unix_socket)
myapp | FileNotFoundError: [Errno 2] No such file or directory
myapp |
myapp | During handling of the above exception, another exception occurred:
myapp |
myapp | Traceback (most recent call last):
myapp | File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
myapp | timeout=timeout
myapp | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
myapp | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
myapp | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 403, in increment
myapp | raise six.reraise(type(error), error, _stacktrace)
myapp | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise
myapp | raise value.with_traceback(tb)
myapp | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
myapp | chunked=chunked,
myapp | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request
myapp | conn.request(method, url, **httplib_request_kw)
myapp | File "/usr/local/lib/python3.7/http/client.py", line 1252, in request
myapp | self._send_request(method, url, body, headers, encode_chunked)
myapp | File "/usr/local/lib/python3.7/http/client.py", line 1298, in _send_request
myapp | self.endheaders(body, encode_chunked=encode_chunked)
myapp | File "/usr/local/lib/python3.7/http/client.py", line 1247, in endheaders
myapp | self._send_output(message_body, encode_chunked=encode_chunked)
myapp | File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output
myapp | self.send(msg)
myapp | File "/usr/local/lib/python3.7/http/client.py", line 966, in send
myapp | self.connect()
myapp | File "/usr/local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 43, in connect
myapp | sock.connect(self.unix_socket)
myapp | urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
myapp |
myapp | During handling of the above exception, another exception occurred:
myapp |
myapp | Traceback (most recent call last):
myapp | File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 207, in _retrieve_server_version
myapp | return self.version(api_version=False)["ApiVersion"]
myapp | File "/usr/local/lib/python3.7/site-packages/docker/api/daemon.py", line 181, in version
myapp | return self._result(self._get(url), json=True)
myapp | File "/usr/local/lib/python3.7/site-packages/docker/utils/decorators.py", line 46, in inner
myapp | return f(self, *args, **kwargs)
myapp | File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 230, in _get
myapp | return self.get(url, **self._set_request_timeout(kwargs))
myapp | File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 543, in get
myapp | return self.request('GET', url, **kwargs)
myapp | File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
myapp | resp = self.send(prep, **send_kwargs)
myapp | File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
myapp | r = adapter.send(request, **kwargs)
myapp | File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
myapp | raise ConnectionError(err, request=request)
myapp | requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
myapp |
myapp | During handling of the above exception, another exception occurred:
myapp |
myapp | Traceback (most recent call last):
myapp | File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 134, in handle
myapp | self.handle_request(listener, req, client, addr)
myapp | File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 175, in handle_request
myapp | respiter = self.wsgi(environ, resp.start_response)
myapp | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2464, in __call__
myapp | return self.wsgi_app(environ, start_response)
myapp | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2450, in wsgi_app
myapp | response = self.handle_exception(e)
myapp | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1867, in handle_exception
myapp | reraise(exc_type, exc_value, tb)
myapp | File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
myapp | raise value
myapp | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
myapp | response = self.full_dispatch_request()
myapp | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
myapp | rv = self.handle_user_exception(e)
myapp | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
myapp | reraise(exc_type, exc_value, tb)
myapp | File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
myapp | raise value
myapp | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
myapp | rv = self.dispatch_request()
myapp | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
myapp | return self.view_functions[rule.endpoint](**req.view_args)
myapp | File "/usr/local/lib/python3.7/site-packages/flask_classful.py", line 268, in proxy
myapp | response = view(**request.view_args)
myapp | File "/usr/local/lib/python3.7/site-packages/flask_classful.py", line 239, in inner
myapp | return fn(*args, **kwargs)
myapp | File "/app/outlier_detection/api/v1/metric.py", line 35, in add
myapp | response = runtime_flow(data_dict)
myapp | File "/app/outlier_detection/flows/runtime_flow.py", line 68, in runtime_flow
myapp | flow.register(labels=['l1'])
myapp | File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1527, in register
myapp | no_url=no_url,
myapp | File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 662, in register
myapp | serialized_flow = flow.serialize(build=build) # type: Any
myapp | File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1375, in serialize
myapp | storage = self.storage.build() # type: Optional[Storage]
myapp | File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 304, in build
myapp | self._build_image(push=push)
myapp | File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 341, in _build_image
myapp | client = self._get_client()
myapp | File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 531, in _get_client
myapp | base_url=self.base_url, version="auto", tls=self.tls_config
myapp | File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 190, in __init__
myapp | self._version = self._retrieve_server_version()
myapp | File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 215, in _retrieve_server_version
myapp | 'Error while fetching server API version: {0}'.format(e)
myapp | docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
myapp | 192.168.192.1 - - [02/Jul/2020:00:43:18 +0530] "POST /api/v1/metric/add/ HTTP/1.1" 500 0 "-" "-" in 10929µs
josh
07/01/2020, 7:19 PMDocker
storage has a base_url
kwarg in which you can pass in the url of a Docker daemon in order to build the storage.myapp
container to achieve this https://medium.com/better-programming/about-var-run-docker-sock-3bfd276e12fdVikram Iyer
07/01/2020, 7:22 PMjosh
07/01/2020, 7:23 PMVikram Iyer
07/01/2020, 7:26 PMjosh
07/01/2020, 7:33 PMbase_url
/ etc.). Otherwise you could look into using a different storage option in conjunction with your docker-based agent https://docs.prefect.io/orchestration/execution/storage_options.html#non-docker-storage-for-containerized-environmentsVikram Iyer
07/01/2020, 7:43 PMjosh
07/01/2020, 7:43 PMVikram Iyer
07/02/2020, 7:29 AMMartin
07/02/2020, 5:18 PMVikram Iyer
07/02/2020, 5:57 PMbuild: .
in docker-compose to say that both myapp and agent should have the current dir built into them. As a final step, make sure to mount the code directory inside both of them.
Does that answer your question?Martin
07/03/2020, 6:51 AMVikram Iyer
07/03/2020, 3:18 PMmyapp:
build: .
command: "command to run your application/service"
networks:
- prefect-server
environment:
PREFECT__BACKEND: server
PREFECT__SERVER__ENDPOINT: <http://apollo:4200>
PREFECT__SERVER__GRAPHQL_URL: <http://apollo:4200/graphql>
agent:
build: .
command: "prefect agent start --label anylabel"
networks:
- prefect-server
environment:
PREFECT__SERVER__GRAPHQL_URL: <http://apollo:4200/graphql>
PREFECT__SERVER__ENDPOINT: <http://apollo:4200>
restart: "always"
Martin
07/06/2020, 7:06 AM