I'm running ~8000 tasks, 1000 concurrently (via mo...
# ask-community
b
I'm running ~8000 tasks, 1000 concurrently (via modal) over the course of 20 minutes, and a handful failed on
database is locked
(full error below) Just letting the prefect team know, not sure if there's anything to be done there.
Copy code
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: INSERT INTO task_run (flow_run_id, task_key, dynamic_key, flow_run_run_count, empirical_policy, task_inputs, tags, labels, name, run_count, expected_start_time, total_run_time, id, created, updated) VALUES (:flow_run_id, :task_key, :dynamic_key, :flow_run_run_count, :empirical_policy, :task_inputs, :tags, :labels, :name, :run_count, :expected_start_time, :total_run_time, :id, :created, :updated) ON CONFLICT (id) DO UPDATE SET flow_run_id = :param_1, task_key = :param_2, dynamic_key = :param_3, flow_run_run_count = :param_4, empirical_policy = :param_5, task_inputs = :param_6, tags = :param_7, labels = :param_8, name = :param_9, run_count = :param_10, expected_start_time = :param_11, total_run_time = :param_12, id = :param_13, updated = :param_14 WHERE task_run.state_timestamp < :state_timestamp_1]
[parameters: {'flow_run_id': None, 'task_key': 'create_flagged_objections-846c687b', 'dynamic_key': 'create_flagged_objections-846c687b-40c67358f2794b4f931badbbf3319531', 'flow_run_run_count': 0, 'empirical_policy': '{"max_retries": 0, "retry_delay_seconds": 0.0, "retries": 0, "retry_delay": 0, "retry_jitter_factor": null}', 'task_inputs': '{"project_id": [], "transcript": [], "objections": [], "type": [], "agent_name": []}', 'tags': '[]', 'labels': '{}', 'name': 'create_flagged_objections', 'run_count': 0, 'expected_start_time': '2025-08-01 22:16:21.968197', 'total_run_time': '1970-01-01 00:00:00.000000', 'id': '7f2f34d0-c402-410b-b37f-006e7a113f30', 'created': '2025-08-01 22:22:52.647446', 'updated': '2025-08-01 22:22:52.648611', 'param_1': None, 'param_2': 'create_flagged_objections-846c687b', 'param_3': 'create_flagged_objections-846c687b-40c67358f2794b4f931badbbf3319531', 'param_4': 0, 'param_5': '{"max_retries": 0, "retry_delay_seconds": 0.0, "retries": 0, "retry_delay": 0, "retry_jitter_factor": null}', 'param_6': '{"project_id": [], "transcript": [], "objections": [], "type": [], "agent_name": []}', 'param_7': '[]', 'param_8': '{}', 'param_9': 'create_flagged_objections', 'param_10': 0, 'param_11': '2025-08-01 22:16:21.968197', 'param_12': '1970-01-01 00:00:00.000000', 'param_13': '7f2f34d0-c402-410b-b37f-006e7a113f30', 'param_14': '2025-08-01 22:22:52.647691', 'state_timestamp_1': '2025-08-01 22:16:21.968197'}]
(Background on this error at: <https://sqlalche.me/e/20/e3q8>)
22:23:53.104 | ERROR   | prefect.server.utilities.messaging.memory - Failed in consume_loop
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1961, in _exec_single_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 944, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 177, in execute
    self._adapt_connection._handle_exception(error)
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 337, in _handle_exception
    raise error
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 159, in execute
    self.await_(_cursor.execute(operation, parameters))
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
    return current.parent.switch(awaitable)  # type: ignore[no-any-return,attr-defined] # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
    value = await result
            ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiosqlite/cursor.py", line 40, in execute
    await self._execute(self._cursor.execute, sql, parameters)
  File "/usr/local/lib/python3.11/site-packages/aiosqlite/cursor.py", line 32, in _execute
    return await self._conn._execute(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiosqlite/core.py", line 122, in _execute
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiosqlite/core.py", line 105, in run
    result = function()
             ^^^^^^^^^^
sqlite3.OperationalError: database is locked
j
Hey, just double checking, youre intending to run a local prefect installation here?
(If so, yes this is a normal consequence of a lot of concurrent operations against the sqlite backend)
b
Yea it’s not meant to be local. Something is wrong that I’m not connecting to cloud, thanks!
j
most likely culprit is PREFECT_API_URL not being set. And np!
❤️ 1
j
does this config is persistent? prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api