https://prefect.io logo
Join Slack
Powered by
Hello, I have a very strange issue. I am running P...
# ask-community
p
Hello, I have a very strange issue. I am running Prefect 2.10.10 on Azure AKS using a managed postgres db and I am getting a connection error. I thought it was dns related but I can resolve the db hostname from a pod in the same namespace. I can also connect with psql. I am passing the db info using 
PREFECT_API_DATABASE_CONNECTION_URL
env variable. Any ideas? Thank you!
✅ 1
Copy code
___ ___ ___ ___ ___ ___ _____ 
| _ \ _ \ __| __| __/ __|_   _| 
|  _/   / _|| _|| _| (__  | |  
|_| |_|_\___|_| |___\___| |_|  
Configure Prefect to communicate with the server with:
    prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>
View the API reference documentation at <http://0.0.0.0:4200/docs>
Check out the dashboard at <http://0.0.0.0:4200>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/server/api/server.py", line 506, in lifespan
    await run_migrations()
  File "/usr/local/lib/python3.8/site-packages/prefect/server/api/server.py", line 429, in run_migrations
    await db.create_db()
  File "/usr/local/lib/python3.8/site-packages/prefect/server/database/interface.py", line 55, in create_db
    await self.run_migrations_upgrade()
  File "/usr/local/lib/python3.8/site-packages/prefect/server/database/interface.py", line 63, in run_migrations_upgrade
    await run_sync_in_worker_thread(alembic_upgrade)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.8/site-packages/prefect/server/database/alembic_commands.py", line 24, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/server/database/alembic_commands.py", line 53, in alembic_upgrade
    alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
  File "/usr/local/lib/python3.8/site-packages/alembic/command.py", line 385, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 582, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/prefect/server/database/migrations/env.py", line 147, in <module>
    apply_migrations()
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 248, in coroutine_wrapper
    return run_async_from_worker_thread(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 177, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/usr/local/lib/python3.8/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/site-packages/prefect/server/database/migrations/env.py", line 140, in apply_migrations
    async with engine.connect() as connection:
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/ext/asyncio/base.py", line 66, in __aenter__
    return await self.start(is_ctxmanager=True)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/ext/asyncio/engine.py", line 157, in start
    await (greenlet_spawn(self.sync_engine.connect))
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 126, in greenlet_spawn
    result = context.throw(*sys.exc_info())
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/future/engine.py", line 406, in connect
    return super(Engine, self).connect()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3320, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3399, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3366, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 327, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 894, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 493, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 273, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 388, in __init__
    self.__connect()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 691, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 686, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/create.py", line 574, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 598, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 780, in connect
    await_only(self.asyncpg.connect(*arg, **kw)),
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
    return current.driver.switch(awaitable)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
    value = await result
  File "/usr/local/lib/python3.8/site-packages/asyncpg/connection.py", line 2092, in connect
    return await connect_utils._connect(
  File "/usr/local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 895, in _connect
    raise last_error
  File "/usr/local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 881, in _connect
    return await _connect_addr(
  File "/usr/local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 773, in _connect_addr
    return await __connect_addr(params, timeout, True, *args)
  File "/usr/local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 825, in __connect_addr
    tr, pr = await compat.wait_for(connector, timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/asyncpg/compat.py", line 56, in wait_for
    return await asyncio.wait_for(fut, timeout)
  File "/usr/local/lib/python3.8/asyncio/tasks.py", line 494, in wait_for
    return fut.result()
  File "/usr/local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 684, in _create_ssl_connection
    tr, pr = await loop.create_connection(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 986, in create_connection
    infos = await self._ensure_resolved(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1365, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 825, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/starlette/datastructures.py", line 702, in __getattr__
    return self._state[key]
KeyError: 'services'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 677, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/local/lib/python3.8/contextlib.py", line 171, in __aenter__
    return await self.gen.__anext__()
  File "/usr/local/lib/python3.8/site-packages/prefect/server/api/server.py", line 511, in lifespan
    await stop_services()
  File "/usr/local/lib/python3.8/site-packages/prefect/server/api/server.py", line 493, in stop_services
    if app.state.services:
  File "/usr/local/lib/python3.8/site-packages/starlette/datastructures.py", line 705, in __getattr__
    raise AttributeError(message.format(self.__class__.__name__, key))
AttributeError: 'State' object has no attribute 'services'
Application startup failed. Exiting.
Server stopped!
z
Can you connect to the database using SQLAlchemy?
Maybe like…
Copy code
from prefect.server.database.configurations import AsyncPostgresConfiguration
from prefect.settings import PREFECT_API_DATABASE_CONNECTION_URL
import asyncio


async def main():
    print(PREFECT_API_DATABASE_CONNECTION_URL.value())
    config = AsyncPostgresConfiguration(PREFECT_API_DATABASE_CONNECTION_URL.value())
    engine = await config.engine()
    session = await config.session(engine)
    print(session)


asyncio.run(main())
p
it seems to be able to connect
Copy code
$ python
Python 3.8.16 (default, Mar 23 2023, 04:48:11)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from prefect.server.database.configurations import AsyncPostgresConfiguration
>>> from prefect.settings import PREFECT_API_DATABASE_CONNECTION_URL
>>> import asyncio
>>>
>>> async def main():
...     print(PREFECT_API_DATABASE_CONNECTION_URL.value())
...     config = AsyncPostgresConfiguration(PREFECT_API_DATABASE_CONNECTION_URL.value())
...     engine = await config.engine()
...     session = await config.session(engine)
...     print(session)
...
>>> asyncio.run(main())
<postgresql+asyncpg://XXXXX:YYYY@ZZZZZZZZZ:5432/WWWWW>
<sqlalchemy.ext.asyncio.session.AsyncSession object at 0x7fa0f4ce1340>
>>>
z
hm weird I’m worried that didn’t actually connect
What if you add a
Copy code
async with engine.connect() as connection:
    pass
p
adding the 
with engine..
I was able to reproduce the issue. It turned out to be a k8s dns issue and once i fixed it I had another issue with the password containing special chars
thanks a lot for your help!
z
Good to hear 🙂
15 Views
Open in Slack
PreviousNext
Powered by