Daniel
01/30/2025, 11:01 AM# Generic metadata about this project
name: amy-doc-ingest-server
prefect-version: 2.19.1
# build section allows you to manage and build docker images
build:
- prefect_docker.deployments.steps.build_docker_image:
id: build_image
requires: prefect-docker>=0.3.1
image_name: europe-west2-docker.pkg.dev/amygdadss/prefect-repo/doc-ingest-pipeline
tag: latest
dockerfile: ./Dockerfile
platform: linux/amd64
# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_docker.deployments.steps.push_docker_image:
requires: prefect-docker>=0.3.1
image_name: '{{ build_image.image_name }}'
tag: '{{ build_image.tag }}'
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.set_working_directory:
directory: /opt/prefect/amy-doc-ingest-server
# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: doc-ingest-pipeline-dev
version:
tags: ["dev"]
description:
entrypoint: flows/doc_ingest_flow.py:process_document
parameters:
doc_file_path: ["amy-app-v1-docs/atl/Class-180-vol-1-small-TEST/Class-180-vol-1-small-TEST.pdf"]
work_pool:
name: cloud-run-work-pool
work_queue_name: default
job_variables:
image: '{{ build_image.image }}'
image_pull_policy: 'Never'
schedules: []
But keep getting this error:
% prefect deploy --name doc-ingest-pipeline-dev --prefect-file prefect_dev.yaml
Running deployment build steps...
> Running build_docker_image step...
Traceback (most recent call last):
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/requests/adapters.py", line 633, in send
conn = self.get_connection_with_tls_context(
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/requests/adapters.py", line 489, in get_connection_with_tls_context
conn = self.poolmanager.connection_from_host(
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/urllib3/poolmanager.py", line 303, in connection_from_host
return self.connection_from_context(request_context)
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/urllib3/poolmanager.py", line 325, in connection_from_context
raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/docker/api/client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/requests/adapters.py", line 637, in send
raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Not supported URL scheme http+docker
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/prefect/utilities/dockerutils.py", line 110, in docker_client
client = docker.DockerClient.from_env()
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/docker/client.py", line 96, in from_env
return cls(
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/docker/client.py", line 45, in __init__
self.api = APIClient(*args, **kwargs)
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/docker/api/client.py", line 197, in __init__
self._version = self._retrieve_server_version()
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps
step_output = await run_step(step, upstream_outputs)
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 125, in run_step
result = await from_async.call_soon_in_new_thread(
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
return await asyncio.wrap_future(self.future)
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
result = self.fn(*self.args, **self.kwargs)
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/prefect_docker/deployments/steps.py", line 188, in build_docker_image
with docker_client() as client:
File "/usr/local/Cellar/python@3.10/3.10.14_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/prefect/utilities/dockerutils.py", line 114, in docker_client
raise RuntimeError(
RuntimeError: This error is often thrown because Docker is not running. Please ensure Docker is running.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/dansimons/amygda/amy-document-ingest/menv/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 42, in wrapper
return fn(*args, **kwargs)
I am using prefect v2.19.9 and docker is running using orb stack. I have a very similar setup for a different docker project that uses same docker through orb stack, same prefect version and it works fine. I can’t seem to work out why this is occurring in this repo. Any help would be much appreciated!Daniel
01/30/2025, 11:44 AM