I upgraded from `2.8.5` to `latest` but now the pr...
# best-practicesr
Richard Alexander
03/07/2024, 2:52 PMI upgraded from
2.8.5latest✅ 1
Richard Alexander
03/07/2024, 2:53 PM Copy code
___ ___ ___ ___ ___ ___ _____
| _ \ _ \ __| __| __/ __|_ _|
| _/ / _|| _|| _| (__ | |
|_| |_|_\___|_| |___\___| |_|
Configure Prefect to communicate with the server with:
prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>
View the API reference documentation at <http://127.0.0.1:4200/docs>
Check out the dashboard at <http://127.0.0.1:4200>
14:15:39.275 | ERROR | uvicorn.error - Traceback (most recent call last):
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/_vendor/starlette/routing.py", line 736, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/server/api/server.py", line 594, in lifespan
await run_migrations()
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/server/api/server.py", line 517, in run_migrations
await db.create_db()
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/server/database/interface.py", line 56, in create_db
await self.run_migrations_upgrade()
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/server/database/interface.py", line 64, in run_migrations_upgrade
await run_sync_in_worker_thread(alembic_upgrade)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 95, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/server/database/alembic_commands.py", line 24, in wrapper
return fn(*args, **kwargs)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/server/database/alembic_commands.py", line 53, in alembic_upgrade
alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/alembic/command.py", line 322, in upgrade
script.run_env()
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
util.load_python_file(self.dir, "env.py")
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
module = load_module_py(module_id, path)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/server/database/migrations/env.py", line 175, in <module>
apply_migrations()
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 247, in coroutine_wrapper
return run_async_from_worker_thread(async_fn, *args, **kwargs)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 181, in run_async_from_worker_thread
return anyio.from_thread.run(call)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/anyio/from_thread.py", line 47, in run
return asynclib.run_async_from_thread(func, *args)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 906, in run_async_from_thread
return f.result()
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/concurrent/futures/_base.py", line 446, in result
return self.__get_result()
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/server/database/migrations/env.py", line 169, in apply_migrations
await connection.run_sync(do_run_migrations)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/engine.py", line 548, in run_sync
return await greenlet_spawn(fn, conn, *arg, **kw)Richard Alexander
03/07/2024, 2:53 PM Copy code
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 128, in greenlet_spawn
result = context.switch(value)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/server/database/migrations/env.py", line 137, in do_run_migrations
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/alembic/runtime/environment.py", line 853, in run_migrations
self.get_context().run_migrations(**kw)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/alembic/runtime/migration.py", line 623, in run_migrations
step.migration_fn(**kw)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/prefect/server/database/migrations/versions/postgresql/2023_04_04_132534_3bf47e3ce2dd_add_i
op.create_index(
File "<string>", line 8, in create_index
File "<string>", line 3, in create_index
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/alembic/operations/ops.py", line 966, in create_index
return operations.invoke(op)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/alembic/operations/base.py", line 399, in invoke
return fn(self, operation)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/alembic/operations/toimpl.py", line 99, in create_index
operations.impl.create_index(idx)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/alembic/ddl/postgresql.py", line 84, in create_index
self._exec(CreateIndex(index))
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/alembic/ddl/impl.py", line 195, in _exec
return conn.execute(construct, multiparams)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/future/engine.py", line 280, in execute
return self._execute_20(
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
return connection._execute_ddl(
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1472, in _execute_ddl
ret = self._execute_context(
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
self._handle_dbapi_exception(
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2128, in _handle_dbapi_exception
util.raise_(exc_info[1], with_traceback=exc_info[2])
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 479, in execute
self._adapt_connection.await_(
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
return current.driver.switch(awaitable)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
value = await result
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 454, in _prepare_and_execute
self._handle_exception(error)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 389, in _handle_exception
self._adapt_connection._handle_exception(error)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
raise error
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 442, in _prepare_and_execute
self._rows = await prepared_stmt.fetch(*parameters)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/asyncpg/prepared_stmt.py", line 176, in fetch
data = await self.__bind_execute(args, 0, timeout)
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/asyncpg/prepared_stmt.py", line 241, in __bind_execute
data, status, _ = await self.__do_execute(
File "/home/ubuntu/miniconda3/envs/prefect2/lib/python3.10/site-packages/asyncpg/prepared_stmt.py", line 230, in __do_execute
return await executor(protocol)
File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
asyncio.exceptions.TimeoutError
14:15:39.283 | ERROR | uvicorn.error - Application startup failed. Exiting.
Server stopped!j
Jake Kaplan
03/07/2024, 3:04 PMHey, I think your server is trying to run migrations on startup to get up to date and timing out on one:
I would try setting
PREFECT_API_DATABASE_TIMEOUTPREFECT_API_DATABASE_CONNECTION_TIMEOUTPREFECT_API_DATABASE_MIGRATE_ON_STARTprefect server database migrations upgrader
Richard Alexander
03/07/2024, 3:05 PMThanks Jake! If I run manually, does it avoid the timeout issue?
j
Jake Kaplan
03/07/2024, 3:06 PMyou'll need to have the timeout env vars configured to beat the timeout. running manually just lets you do that separate from trying to start your server
r
Richard Alexander
03/07/2024, 3:07 PMThere might be another issue when running manually...
j
Jake Kaplan
03/07/2024, 3:07 PMthe issue might be with me... 😅
Jake Kaplan
03/07/2024, 3:07 PMsorry the command is
prefect server database upgrade😄 1
r
Richard Alexander
03/07/2024, 3:08 PMGot it... that command worked. And where should I set those env vars?
j
Jake Kaplan
03/07/2024, 3:09 PMjust wherever you're running the migration command from and you can remove them after.
Copy code
prefect config set PREFECT_API_DATABASE_TIMEOUT=99999 (or seomething) Copy code
prefect config unset PREFECT_API_DATABASE_TIMEOUTr
Richard Alexander
03/07/2024, 3:26 PMYep, that fixed it!
Thanks a ton Jake, you're a lifesaver! 😃🎉
🙌 1
51 Views