After installed Prefect Orion 2.7, I am getting the following exception and I couldn't see any of my existing flow-runs in Orion UI. TypeError: unsupported operand type(s) for -: 'DateTime' and 'NoneType'

Hey @Xavier Babu Can you move the stacktrace into this thread it helps keep the main channel clean

23:06:51.452 | ERROR | uvicorn.error - Exception in ASGI application Traceback (most recent call last): File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi result = await app(self.scope, self.receive, self.send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in call return await self.app(scope, receive, send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/uvicorn/middleware/message_logger.py", line 82, in call raise exc from None File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/uvicorn/middleware/message_logger.py", line 78, in call await self.app(scope, inner_receive, inner_send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/fastapi/applications.py", line 261, in call await super().__call__(scope, receive, send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/applications.py", line 112, in call await self.middleware_stack(scope, receive, send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/middleware/errors.py", line 181, in call raise exc File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/middleware/errors.py", line 159, in call await self.app(scope, receive, _send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/middleware/cors.py", line 92, in call await self.simple_response(scope, receive, send, request_headers=headers) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/middleware/cors.py", line 147, in simple_response await self.app(scope, receive, send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/exceptions.py", line 82, in call raise exc File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/exceptions.py", line 71, in call await self.app(scope, receive, sender) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call raise e File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call await self.app(scope, receive, send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/routing.py", line 656, in call await route.handle(scope, receive, send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/routing.py", line 408, in handle await self.app(scope, receive, send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/fastapi/applications.py", line 261, in call await super().__call__(scope, receive, send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/applications.py", line 112, in call await self.middleware_stack(scope, receive, send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/middleware/errors.py", line 181, in call raise exc File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/middleware/errors.py", line 159, in call await self.app(scope, receive, _send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/exceptions.py", line 82, in call raise exc File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/exceptions.py", line 71, in call await self.app(scope, receive, sender) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call raise e File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call await self.app(scope, receive, send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/routing.py", line 656, in call await route.handle(scope, receive, send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/routing.py", line 259, in handle await self.app(scope, receive, send) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/starlette/routing.py", line 61, in app response = await func(request) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/orion/utilities/server.py", line 101, in handle_response_scoped_depends response = await default_handler(request) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app raw_response = await run_endpoint_function( File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/fastapi/routing.py", line 160, in run_endpoint_function return await dependant.call(**values) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/orion/api/ui/flow_runs.py", line 74, in read_flow_run_history return [ File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/orion/api/ui/flow_runs.py", line 79, in <listcomp> duration=r.estimated_run_time, File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/sqlalchemy/ext/hybrid.py", line 927, in get return self.fget(instance) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/orion/database/orm_models.py", line 271, in estimated_run_time return self.total_run_time + (pendulum.now() - self.state_timestamp) TypeError: unsupported operand type(s) for -: 'DateTime' and 'NoneType'

Hey @Xavier Babu What OS are you running the Orion Instance on, it's also possible this is a database issue/conflict that came about from updating what version of Orion were you on before?

Linux Prefect Orion 2.4.5 to 2.7

@Xavier Babu is this error coming up from when you're trying to run a flow? This shouldn't be the case but if when you updated prefect are the configuration pointing to this Orion Instance? In that same vein where is this Orion instance hosted. When you run a Flow run or attempt to are you seeing the data populated in the Orion DB, the error is basically indicating that the state data for the flow run doesn't exist

Mason, When I click "Flow Runs" in Prefect UI, getting this exception in my Orion log.

If you check the database do the flow_runs show up, I'm not really an expert on postgres databases so I'm just going based on the error here, @Anna Geller do you know where this error could be coming from?

it looks like it doesn't correctly point to the Postgres DB and instead tries to use sqlite located on a shared drive - @Xavier Babu could you try following this guide? https://discourse.prefect.io/t/scaling-prefect-whats-the-difference-between-ephemeral-and-hosted-api/1990

Based on the log, it was talking to the right DB (postgres). I couldn't even reset and upgrade the DB, I got many drop index failure messages. The only way finally I fixed this issue is, by removing all Prefect Tables and indexes manually and reset the DB with Prefect Orion 2.7.

could be, nice work fixing the issue!