Updated orion to 2 7 11 from 2 7 10 server started crashloop

Question

Updated orion to 2 7 11 from 2 7 10 server started crashlooping turned on debug logs got `sqlalchemy exc OperationalError sqlite3 OperationalError duplicate column name has data` see thread Seems like the culpable migration is in a commit merged last week Downgraded to 2 7 10 got `alembic util exc CommandError Can t locate revision identified by f92143d30c24 ` Managed to solve by running `prefect orion database downgrade r bb38729c471a` from 2 7 11 and downgrading to 2 7 10 worked Tried to reproduce by upgrading to 2 7 11 and starting the server = gt same exception same solution Can t say if this was the original reason for crashlooping or something else left database in a weird state Orion is running on k8s and using sqlite

Q · Answer

``` | | | | | | | | | | || || | | | | | | | | ` | | | | | | | | | | | | | | | | Configure Prefect to communicate with the server with prefect config set PREFECT API URL= View the API reference documentation at Check out the dashboard at Traceback most recent call last File venv lib python3 9 site packages sqlalchemy engine base py line 1900 in execute context self dialect do execute File venv lib python3 9 site packages sqlalchemy engine default py line 736 in do execute cursor execute statement parameters File venv lib python3 9 site packages sqlalchemy dialects sqlite aiosqlite py line 108 in execute self adapt connection handle exception error File venv lib python3 9 site packages sqlalchemy dialects sqlite aiosqlite py line 236 in handle exception raise error File venv lib python3 9 site packages sqlalchemy dialects sqlite aiosqlite py line 90 in execute self await cursor execute operation parameters File venv lib python3 9 site packages sqlalchemy util concurrency py3k py line 68 in await only return current driver switch awaitable File venv lib python3 9 site packages sqlalchemy util concurrency py3k py line 121 in greenlet spawn value = await result File venv lib python3 9 site packages aiosqlite cursor py line 37 in execute await self execute self cursor execute sql parameters File venv lib python3 9 site packages aiosqlite cursor py line 31 in execute return await self conn execute fn args kwargs File venv lib python3 9 site packages aiosqlite core py line 137 in execute return await future File venv lib python3 9 site packages aiosqlite core py line 110 in run result = function sqlite3 OperationalError duplicate column name has data The above exception was the direct cause of the following exception Traceback most recent call last File venv lib python3 9 site packages starlette routing py line 671 in lifespan async with self lifespan context app File venv lib python3 9 site packages starlette routing py line 566 in aenter await self router startup File venv lib python3 9 site packages starlette routing py line 648 in startup await handler File venv lib python3 9 site packages prefect orion api server py line 348 in run migrations await db create db File venv lib python3 9 site packages prefect orion database interface py line 55 in create db await self run migrations upgrade File venv lib python3 9 site packages prefect orion database interface py line 63 in run migrations upgrade await run sync in worker thread alembic upgrade File venv lib python3 9 site packages prefect utilities asyncutils py line 91 in run sync in worker thread return await anyio to thread run sync File venv lib python3 9 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File venv lib python3 9 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File venv lib python3 9 site packages anyio backends asyncio py line 867 in run result = context run func args File venv lib python3 9 site packages prefect orion database alembic commands py line 24 in wrapper return fn args kwargs File venv lib python3 9 site packages prefect orion database alembic commands py line 53 in alembic upgrade alembic command upgrade alembic config revision sql=dry run File venv lib python3 9 site packages alembic command py line 378 in upgrade script run env File venv lib python3 9 site packages alembic script base py line 569 in run env util load python file self dir env py File venv lib python3 9 site packages alembic util pyfiles py line 94 in load python file module = load module py module id path File venv lib python3 9 site packages alembic util pyfiles py line 110 in load module py spec loader exec module module type ignore File lt frozen importlib bootstrap external gt line 850 in exec module File lt frozen importlib bootstrap gt line 228 in call with frames removed File venv lib python3 9 site packages prefect orion database migrations env py line 147 in lt module gt apply migrations File venv lib python3 9 site packages prefect utilities asyncutils py line 226 in coroutine wrapper return run async from worker thread async fn args kwargs File venv lib python3 9 site packages prefect utilities asyncutils py line 177 in run async from worker thread return anyio from thread run call File venv lib python3 9 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File venv lib python3 9 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 9 concurrent futures base py line 445 in result return self get result File usr local lib python3 9 concurrent futures base py line 390 in get result raise self exception File venv lib python3 9 site packages prefect orion database migrations env py line 141 in apply migrations await connection run sync do run migrations File venv lib python3 9 site packages sqlalchemy ext asyncio engine py line 548 in run sync return await greenlet spawn fn conn arg kw File venv lib python3 9 site packages sqlalchemy util concurrency py3k py line 128 in greenlet spawn result = context switch value File venv lib python3 9 site packages prefect orion database migrations env py line 129 in do run migrations context run migrations File lt string gt line 8 in run migrations File venv lib python3 9 site packages alembic runtime environment py line 867 in run migrations self get context run migrations kw File venv lib python3 9 site packages alembic runtime migration py line 624 in run migrations step migration fn kw File venv lib python3 9 site packages prefect orion database migrations versions sqlite 2023 01 12 000043 f92143d30c25 create migration index py line 23 in upgrade batch op create index File usr local lib python3 9 contextlib py line 126 in exit next self gen File venv lib python3 9 site packages alembic operations base py line 383 in batch alter table impl flush File venv lib python3 9 site packages alembic operations batch py line 111 in flush fn arg kw File venv lib python3 9 site packages alembic ddl impl py line 322 in add column self exec base AddColumn table name column schema=schema File venv lib python3 9 site packages alembic ddl impl py line 193 in exec return conn execute type ignore call overload File venv lib python3 9 site packages sqlalchemy future engine py line 280 in execute return self execute 20 File venv lib python3 9 site packages sqlalchemy engine base py line 1705 in execute 20 return meth self args 10style kwargs 10style execution options File venv lib python3 9 site packages sqlalchemy sql ddl py line 80 in execute on connection return connection execute ddl File venv lib python3 9 site packages sqlalchemy engine base py line 1472 in execute ddl ret = self execute context File venv lib python3 9 site packages sqlalchemy engine base py line 1943 in execute context self handle dbapi exception File venv lib python3 9 site packages sqlalchemy engine base py line 2124 in handle dbapi exception util raise File venv lib python3 9 site packages sqlalchemy util compat py line 211 in raise raise exception File venv lib python3 9 site packages sqlalchemy engine base py line 1900 in execute context self dialect do execute File venv lib python3 9 site packages sqlalchemy engine default py line 736 in do execute cursor execute statement parameters File venv lib python3 9 site packages sqlalchemy dialects sqlite aiosqlite py line 108 in execute self adapt connection handle exception error File venv lib python3 9 site packages sqlalchemy dialects sqlite aiosqlite py line 236 in handle exception raise error File venv lib python3 9 site packages sqlalchemy dialects sqlite aiosqlite py line 90 in execute self await cursor execute operation parameters File venv lib python3 9 site packages sqlalchemy util concurrency py3k py line 68 in await only return current driver switch awaitable File venv lib python3 9 site packages sqlalchemy util concurrency py3k py line 121 in greenlet spawn value = await result File venv lib python3 9 site packages aiosqlite cursor py line 37 in execute await self execute self cursor execute sql parameters File venv lib python3 9 site packages aiosqlite cursor py line 31 in execute return await self conn execute fn args kwargs File venv lib python3 9 site packages aiosqlite core py line 137 in execute return await future File venv lib python3 9 site packages aiosqlite core py line 110 in run result = function sqlalchemy exc OperationalError sqlite3 OperationalError duplicate column name has data SQL ALTER TABLE flow run state ADD COLUMN has data BOOLEAN Background on this error at Application startup failed Exiting Orion stopped ```

Q · Answer