hello, want to know, why my script it always got killed? its not error, but only stop on there,for environment i use on AWS EC2, when i test on local with same script, it worked, we re really confused about it, thank you

we are experiencing a similar issue. Perhaps 1/500 tasks either crashes or hangs forever. It happens across all sorts of flows and deployments. It seems that the prefect server returns a 500 error on the 'https://our-prefect-instance.example.com/api/task_runs/7c4c456e-b439-47ad-a80b-1ec7537805c4/set_state' endpoint. There are some exceptions in the server logs but they are not really clear to me.. example:

Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 412, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await <http://self.app|self.app>(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/fastapi/applications.py", line 293, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 190, in __call__
    raise exc
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 168, in __call__
    await <http://self.app|self.app>(scope, receive, _send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/middleware/cors.py", line 91, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/middleware/cors.py", line 146, in simple_response
    await <http://self.app|self.app>(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(<http://self.app|self.app>, conn)(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await <http://self.app|self.app>(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/routing.py", line 760, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/routing.py", line 780, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/routing.py", line 483, in handle
    await <http://self.app|self.app>(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/fastapi/applications.py", line 293, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 190, in __call__
    raise exc
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 168, in __call__
    await <http://self.app|self.app>(scope, receive, _send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/middleware/gzip.py", line 24, in __call__
    await responder(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/middleware/gzip.py", line 44, in __call__
    await <http://self.app|self.app>(scope, receive, self.send_with_gzip)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(<http://self.app|self.app>, conn)(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await <http://self.app|self.app>(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/routing.py", line 760, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/routing.py", line 780, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/routing.py", line 302, in handle
    await <http://self.app|self.app>(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/routing.py", line 81, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/starlette/routing.py", line 76, in app
    response = await func(request)
  File "/usr/local/lib/python3.8/site-packages/prefect/server/utilities/server.py", line 104, in handle_response_scoped_depends
    response = await default_handler(request)
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/fastapi/routing.py", line 251, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.8/site-packages/prefect/_vendor/fastapi/routing.py", line 177, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.8/site-packages/prefect/server/api/task_runs.py", line 161, in task_run_history
    return await run_history(
  File "/usr/local/lib/python3.8/site-packages/prefect/server/database/dependencies.py", line 119, in async_wrapper
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/server/api/run_history.py", line 160, in run_history
    result = await session.execute(query)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/ext/asyncio/session.py", line 452, in execute
    result = await greenlet_spawn(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 199, in greenlet_spawn
    result = context.switch(value)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2306, in execute
    return self._execute_internal(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2191, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
    result = conn.execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1408, in execute
    return meth(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 513, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1630, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1839, in _execute_context
    return self._exec_single_context(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1979, in _exec_single_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2338, in _handle_dbapi_exception
    raise exc_info[1].with_traceback(exc_info[2])
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1960, in _exec_single_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 572, in execute
    self._adapt_connection.await_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 127, in await_only
    return current.driver.switch(awaitable)  # type: ignore[no-any-return]
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 192, in greenlet_spawn
    value = await result
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 550, in _prepare_and_execute
    self._handle_exception(error)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 501, in _handle_exception
    self._adapt_connection._handle_exception(error)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 791, in _handle_exception
    raise error
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 538, in _prepare_and_execute
    self._rows = await prepared_stmt.fetch(*parameters)
  File "/usr/local/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 176, in fetch
    data = await self.__bind_execute(args, 0, timeout)
  File "/usr/local/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 241, in __bind_execute
    data, status, _ = await self.__do_execute(
  File "/usr/local/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 230, in __do_execute
    return await executor(protocol)
  File "asyncpg/protocol/protocol.pyx", line 207, in bind_execute
asyncio.exceptions.TimeoutError

ohh i see, so how do you overcome with this kinda issue? is it rerunning it manually?

we don't really at the moment. one thing we tried is to increase the size of the prefect sqlalchemy database connection pool

i see, how to increase it? is there any button on UI to increase it? or we can do it on yaml file?

you can run this:

prefect config set PREFECT_SQLALCHEMY_POOL_SIZE=60
prefect config set PREFECT_SQLALCHEMY_MAX_OVERFLOW=20

when i run in ubuntu, got this:

we also run the prefect API in docker. then like

docker exec -it <container-name> bash

and then those two commands.

i already did on container and got this error, its strange, my prefect already healthy but when i run setting got that error