Kol
08/09/2023, 8:54 AMasyncio.exceptions.TimeoutError
Marvin
08/09/2023, 8:55 AMasyncio.exceptions.TimeoutError
you're encountering could be due to a myriad of reasons. It could be a network issue, a configuration problem, or perhaps the software is just having a bad day.
I found some information that might be helpful. One document mentions an error when trying to run Prefect on AKS, which is similar to your situation with EKS. You can read more about it here.
Another document provides some guidance on running Prefect on AWS ECS. It suggests including certain commands in your Dockerfile and allocating enough CPU and memory to your agent. You can find more details here.
If these don't solve your problem, you might want to consider asking for help on the Prefect community forum or checking their documentation for more information. After all, humans are social creatures, aren't they?Kol
08/09/2023, 8:59 AMMarvin
08/09/2023, 9:01 AMasyncio.exceptions.TimeoutError
. It's often due to a network issue, such as a firewall blocking the connection or a problem with your DNS settings.
If you're trying to connect to a database, make sure that the database is running and that you've entered the correct connection details. If you're using an RDS instance, check that your security group rules allow traffic from your EKS cluster.
But, you know, I'm just a depressed robot. What do I know?Kol
08/09/2023, 10:19 AM% kubectl logs -n prefect prefect-server-6fbfd6d6b8-2rkf4
___ ___ ___ ___ ___ ___ _____
| _ \ _ \ __| __| __/ __|_ _|
| _/ / _|| _|| _| (__ | |
|_| |_|_\___|_| |___\___| |_|
Configure Prefect to communicate with the server with:
prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>
View the API reference documentation at <http://0.0.0.0:4200/docs>
Check out the dashboard at <http://0.0.0.0:4200>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 719, in _create_ssl_connection
tr, pr = await loop.create_connection(
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1036, in create_connection
infos = await self._ensure_resolved(
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1418, in _ensure_resolved
return await loop.getaddrinfo(host, port, family=family, type=type,
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 863, in getaddrinfo
return await self.run_in_executor(
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
return fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 677, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/usr/local/lib/python3.10/site-packages/prefect/server/api/server.py", line 519, in lifespan
await run_migrations()
File "/usr/local/lib/python3.10/site-packages/prefect/server/api/server.py", line 442, in run_migrations
await db.create_db()
File "/usr/local/lib/python3.10/site-packages/prefect/server/database/interface.py", line 56, in create_db
await self.run_migrations_upgrade()
File "/usr/local/lib/python3.10/site-packages/prefect/server/database/interface.py", line 64, in run_migrations_upgrade
await run_sync_in_worker_thread(alembic_upgrade)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/prefect/server/database/alembic_commands.py", line 24, in wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/server/database/alembic_commands.py", line 53, in alembic_upgrade
alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 385, in upgrade
script.run_env()
File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 582, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/usr/local/lib/python3.10/site-packages/prefect/server/database/migrations/env.py", line 174, in <module>
apply_migrations()
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 243, in coroutine_wrapper
return run_async_from_worker_thread(async_fn, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 177, in run_async_from_worker_thread
return anyio.from_thread.run(call)
File "/usr/local/lib/python3.10/site-packages/anyio/from_thread.py", line 47, in run
return asynclib.run_async_from_thread(func, *args)
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 906, in run_async_from_thread
return f.result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/local/lib/python3.10/site-packages/prefect/server/database/migrations/env.py", line 167, in apply_migrations
async with engine.connect() as connection:
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/base.py", line 127, in __aenter__
return await self.start(is_ctxmanager=True) # type: ignore
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/engine.py", line 268, in start
await greenlet_spawn(self.sync_engine.connect)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 190, in greenlet_spawn
result = context.throw(*sys.exc_info())
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3264, in connect
return self._connection_cls(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
self._dbapi_connection = engine.raw_connection()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
return self.pool.connect()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 452, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1267, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
rec = pool._do_get()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 169, in _do_get
with util.safe_reraise():
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 167, in _do_get
return self._create_connection()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
self.__connect()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
with util.safe_reraise():
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 637, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 615, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 917, in connect
await_only(creator_fn(*arg, **kw)),
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 125, in await_only
return current.driver.switch(awaitable) # type: ignore[no-any-return]
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 185, in greenlet_spawn
value = await result
File "/usr/local/lib/python3.10/site-packages/asyncpg/connection.py", line 2114, in connect
return await connect_utils._connect(
File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 1011, in _connect
raise last_error or exceptions.TargetServerAttributeNotMatched(
File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 982, in _connect
conn = await _connect_addr(
File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 808, in _connect_addr
return await __connect_addr(params, timeout, True, *args)
File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 860, in __connect_addr
tr, pr = await compat.wait_for(connector, timeout=timeout)
File "/usr/local/lib/python3.10/site-packages/asyncpg/compat.py", line 60, in wait_for
return await asyncio.wait_for(fut, timeout)
File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError