https://prefect.io logo
Join Slack
Powered by
Any reason why we'd be seeing this? ```.venv/lib/...
# marvin-ai
j
Any reason why we'd be seeing this?
Copy code
.venv/lib/python3.11/site-packages/alembic/script/base.py", line 286, in _catch_revision_errors
    raise util.CommandError(resolution) from re
alembic.util.exc.CommandError: Can't locate revision identified by '4ad4658cbefe'
when running a cf.task, I sometimes see Controflow migrations attempting to be run, which is super super bizarre and very concerning
Full trace
Copy code
Traceback (most recent call last):
  File ".venv/lib/python3.11/site-packages/starlette/routing.py", line 693, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File ".pyenv/versions/3.11.9/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/prefect/server/api/server.py", line 600, in lifespan
    await run_migrations()
  File ".venv/lib/python3.11/site-packages/prefect/server/api/server.py", line 509, in run_migrations
    await db.create_db()
  File ".venv/lib/python3.11/site-packages/prefect/server/database/interface.py", line 56, in create_db
    await self.run_migrations_upgrade()
  File ".venv/lib/python3.11/site-packages/prefect/server/database/interface.py", line 64, in run_migrations_upgrade
    await run_sync_in_worker_thread(alembic_upgrade)
  File ".venv/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 269, in run_sync_in_worker_thread
    result = await anyio.to_thread.run_sync(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2441, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 943, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 279, in call_with_mark
    return call()
           ^^^^^^
  File ".venv/lib/python3.11/site-packages/prefect/server/database/alembic_commands.py", line 28, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/prefect/server/database/alembic_commands.py", line 64, in alembic_upgrade
    alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
  File ".venv/lib/python3.11/site-packages/alembic/command.py", line 406, in upgrade
    script.run_env()
  File ".venv/lib/python3.11/site-packages/alembic/script/base.py", line 586, in run_env
    util.load_python_file(self.dir, "env.py")
  File ".venv/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File ".venv/lib/python3.11/site-packages/prefect/server/database/migrations/env.py", line 179, in <module>
    run_async_from_worker_thread(apply_migrations)
  File ".venv/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 290, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/anyio/from_thread.py", line 59, in run
    return async_backend.run_async_from_thread(func, args, token=token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2481, in run_async_from_thread
    return f.result()
           ^^^^^^^^^^
  File ".pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File ".venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2468, in task_wrapper
    return await func(*args)
           ^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/prefect/server/database/migrations/env.py", line 168, in apply_migrations
    await connection.run_sync(do_run_migrations)
  File ".venv/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/engine.py", line 886, in run_sync
    return await greenlet_spawn(
           ^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
    result = context.switch(value)
             ^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/prefect/server/database/migrations/env.py", line 137, in do_run_migrations
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File ".venv/lib/python3.11/site-packages/alembic/runtime/environment.py", line 946, in run_migrations
    self.get_context().run_migrations(**kw)
  File ".venv/lib/python3.11/site-packages/alembic/runtime/migration.py", line 616, in run_migrations
    for step in self._migrations_fn(heads, self):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/alembic/command.py", line 395, in upgrade
    return script._upgrade_revs(revision, rev)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/alembic/script/base.py", line 450, in _upgrade_revs
    with self._catch_revision_errors(
  File ".pyenv/versions/3.11.9/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File ".venv/lib/python3.11/site-packages/alembic/script/base.py", line 286, in _catch_revision_errors
    raise util.CommandError(resolution) from re
alembic.util.exc.CommandError: Can't locate revision identified by '4ad4658cbefe'
c
Isn't that the prefect own migration? I assume your env is messed up
j
Right, I guess question is how?
not sure what would cause it to trigger an alembic migration on a dependency
c
Copy code
➜  ~/.prefect ls -1              
kube.conf
kube.conf.json
memo_store.toml
prefect.db
profiles.toml
profiles.toml.bak
server.pid
storage
if your profile is local, you might wanna try delete that db - make sure to backup though
to answer the "how" we need a magic lamp hehe I doubt that can occur by installing prefect trunk via 
pip
, did you perhaps install an editable local verison?
j
hmm. we just install it via poetry as a dep in our pyproject file
c
try a fresh environment, likely your env is poisoned, hard to figure out how though
j
If you're not running a standalone Prefect server, it starts an in-process server when you run a workflow that attempts to upgrade its own database which is why the migrations are attempted. Almost always, they should silently skip because there's no need to run them. The fact that you're seeing a can't locate revision error is very weird and would only happen if a migration was introduced and then deleted in a subseqeunt error (which would obviously be bad), are you using any of the nightly prefect builds?
is it possible that you have two different envs that might have two different versions of controlflow (and therefore Prefect) installed, which are attempting to access the same sqlite database?
j
hmmm very strange. we're not but now i'm curious to double check. we specify the dep as controlflow = "^0.11.0"
j
hypothetically i could imagine this situation happening if one CF env upgraded the database to include this revision, and then a different CF env with an earlier version of Prefect attempted the migration and failed.
j
its quite possible. hmm. we have a monorepo setup with many different services and libs, so it could maybe end up in that state somehow
j
ok I will raise this to the team and see if we can protect against it at least as an error, if we can figure out how it happened
if you try 
pip install controlflow -U
i wonder if it goes away
or even pip install prefect -U
j
should we be pinning it?
maybe the caret is causing issues?
j
i don't think so
its definitely weird so granted my beliefs might be off, but I think this has to be that one version of Prefect upgraded the db and then a second, lower version of Prefect attempted to access the same db file
question is how
j
ahhh. gotcha. that does make sense logically
j
yeah but also I'm fairly sure we defend against this 🤔 because its unusual but not impossible
let me check internally if this is a known thing
c
just to clarify, I faced this in the past too, though I run an editable install. Having deleted the 
.db
is what had it solved
j
Yeah, this is unpleasant but not uncommon with editable installs because revision IDs change if there are multiple PRs open with migrations (e.g. if your migration is merged before mine, I have to change my revision to maintain linear history, which is usually easiest by just regenerating it with a new ID). But with "proper" releases it should never happen -- only way I can think is if two versions of Prefect are trying to play around with the same db.
👆 1
🎯 1
c
cough veersioneer cough cough
a side note related to packaging, I use 
rye
for workspace management and had to completely repackage the fork I run, ditching 
setup.cfg
which is moving towards deprecation in favor of PEP 518. Been wondering the reason behind Prefect's adoption of legacy setuptools configuration - is it due to a particular requirement, or is migration already planned?
j
I don't know if there's a specific reason for it other than prioritizing a migration over other work
c
hatch and pdm are amazing modern build tools. declarative hooks can be specified for running migrations. Worth taking a look
26 Views
Open in Slack
PreviousNext
Powered by