https://prefect.io logo
Title
g

Georgiana Ogrean

10/10/2022, 6:38 PM
I have a workflow that uses Prefect 2 and is run from inside a Docker container. I am able to run the workflow, bring up the Orion UI, and see the workflow in the UI with the states changing as expected. However, I am stuck at trying to create a deployment, with an error saying (full traceback in 🧵):
ConnectionRefusedError: [Errno 111] Connect call failed ('0.0.0.0', 4200)
The Orion UI comes up just fine when I do not try to build a deployment with:
prefect deployment build /app/my_package/flows/my_python_script.py:my_flow_name --name generate_deployment --tag dev --infra docker-container
Any ideas what I could be missing? More details in the thread.
I have the API_URL properties set with:
prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>
prefect config set PREFECT_ORION_UI_API_URL=http://"${host_ip}":4200/api
Then the Orion UI is started with:
prefect orion start --host=0.0.0.0
And this all works fine as long as I don’t build a deployment.
Below is the full traceback I get when trying to create a deployment:
Traceback (most recent call last):
  File "/app/venv/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
  File "/app/venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
    await get_running_loop().create_connection(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1056, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1041, in create_connection
    sock = await self._connect_sock(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 955, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 502, in sock_connect
    return await fut
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 537, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('0.0.0.0', 4200)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/app/venv/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/app/venv/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 222, in connect_tcp
    raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/app/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/app/venv/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/app/venv/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/app/venv/lib/python3.9/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
    raise exc
  File "/app/venv/lib/python3.9/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
  File "/app/venv/lib/python3.9/site-packages/httpcore/_async/connection.py", line 111, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
  File "/app/venv/lib/python3.9/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
    return await self._backend.connect_tcp(
  File "/app/venv/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/app/venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/venv/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/app/venv/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 201, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/app/venv/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 152, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/app/venv/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/app/venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/app/venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/app/venv/lib/python3.9/site-packages/prefect/cli/deployment.py", line 719, in build
    deployment = await Deployment.build_from_flow(
  File "/app/venv/lib/python3.9/site-packages/prefect/deployments.py", line 679, in build_from_flow
    await deployment.load()
  File "/app/venv/lib/python3.9/site-packages/prefect/deployments.py", line 483, in load
    deployment = await client.read_deployment_by_name(
  File "/app/venv/lib/python3.9/site-packages/prefect/client/orion.py", line 1280, in read_deployment_by_name
    response = await self._client.get(f"/deployments/name/{name}")
  File "/app/venv/lib/python3.9/site-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/app/venv/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/app/venv/lib/python3.9/site-packages/prefect/client/base.py", line 159, in send
    await super().send(*args, **kwargs)
  File "/app/venv/lib/python3.9/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/app/venv/lib/python3.9/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/app/venv/lib/python3.9/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/app/venv/lib/python3.9/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/app/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/app/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
An exception occurred.
The order of the steps executed is: • set up the URL configs • start the Orion UI • create the deployment (this fails) • start the Flask app that has an endpoint which triggers the workflow
Also, I am running Prefect 2.5.0
z

Zanie

10/10/2022, 7:22 PM
You’ll want the API URL to reference the host ip as you do in the
PREFECT_ORION_UI_API_URL
.
👀 1
We even infer
PREFECT_ORION_UI_API_URL
from the
PREFECT_API_URL
if it’s not set.
g

Georgiana Ogrean

10/10/2022, 8:20 PM
Thanks, Michael. I changed
PREFECT_API_URL
and now seeing a different exception when trying to create the deployment:
Traceback (most recent call last):
  File "/app/venv/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/app/venv/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 218, in connect_tcp
    await event.wait()
  File "/app/venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/app/venv/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 164, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
  File "/app/venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1691, in connect_tcp
    await get_running_loop().create_connection(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1041, in create_connection
    sock = await self._connect_sock(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 955, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 502, in sock_connect
    return await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/app/venv/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/app/venv/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
    raise TimeoutError
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/app/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/app/venv/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/app/venv/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/app/venv/lib/python3.9/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
    raise exc
  File "/app/venv/lib/python3.9/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
  File "/app/venv/lib/python3.9/site-packages/httpcore/_async/connection.py", line 111, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
  File "/app/venv/lib/python3.9/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
    return await self._backend.connect_tcp(
  File "/app/venv/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/app/venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ConnectTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/venv/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/app/venv/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 201, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/app/venv/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 152, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/app/venv/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/app/venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/app/venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/app/venv/lib/python3.9/site-packages/prefect/cli/deployment.py", line 719, in build
    deployment = await Deployment.build_from_flow(
  File "/app/venv/lib/python3.9/site-packages/prefect/deployments.py", line 679, in build_from_flow
    await deployment.load()
  File "/app/venv/lib/python3.9/site-packages/prefect/deployments.py", line 483, in load
    deployment = await client.read_deployment_by_name(
  File "/app/venv/lib/python3.9/site-packages/prefect/client/orion.py", line 1280, in read_deployment_by_name
    response = await self._client.get(f"/deployments/name/{name}")
  File "/app/venv/lib/python3.9/site-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/app/venv/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/app/venv/lib/python3.9/site-packages/prefect/client/base.py", line 159, in send
    await super().send(*args, **kwargs)
  File "/app/venv/lib/python3.9/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/app/venv/lib/python3.9/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/app/venv/lib/python3.9/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/app/venv/lib/python3.9/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/app/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/app/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout
An exception occurred.
j

Jeff Hale

10/12/2022, 3:30 AM
What does your code look like?
g

Georgiana Ogrean

10/12/2022, 11:45 PM
I have a Flask app that is run from a
wsgi.py
file in the project directory:
if __name__ == "__main__":
    app.run(host='0.0.0.0', port=9695, use_reloader=False)
The app is created in the
__init__
file of the project directory with:
def init_app():
    """Create a Flask app with an embedded Dash app."""
    app = Flask(__name__, instance_relative_config=False)
    app.config.from_object('config.Config')

    with app.app_context():
        from . import routes

        # Import Dash application
        from .dashboard import init_dashboard
        app = init_dashboard(app)

        return app
so it also brings up a Dash dashboard, if that makes any difference. In the top directory, there is a
Dockerfile
that creates a virtual environment, installs some dependencies, changes the
PATH
environment variable inside the container, and then executes a bunch of commands in a
docker-entrypoint.py
file. So something like:
RUN . venv/bin/activate \
    && poetry install --only main

ENV PATH="/app/venv/bin:$PATH"
ENTRYPOINT ["/app/docker-entrypoint.sh"]
The Orion UI is started from the
docker-entrypoint.sh
file, and that is also where the app is started by executing the code in `wsgi.py`:
#! /bin/bash

# TODO: this is embarrassing, but will sort it out later
host_ip=$(curl <http://checkip.amazonaws.com>)

prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>
prefect config set PREFECT_ORION_UI_API_URL=http://"${host_ip}":4200/api
prefect profile use default

# run the Orion UI in the background
prefect orion start --host=0.0.0.0 &

# TODO: this fails!
prefect deployment build /app/meerkat_statistics/flows/monthly_reports.py:generate_report --name generate_report_deployment --tag dev

python3 wsgi.py
I tried changing
PREFECT_API_URL
to
http://"${host_ip}":4200/api
, but by doing that the workflows triggered/run were no longer showing up in the Orion UI. So I switched that back to
0.0.0.0:4200/api
. With either value of
PREFECT_API_URL
though,
prefect deployment build ...
fails with one of the two errors above.