https://prefect.io logo
Powered by
Title
j

Jarvis Stubblefield

12/07/2022, 12:26 AM
I’m trying to run 
prefect agent -q tenzinga_django_prod
on my server … I’m using a 
systemd
to run. It errors and lets me know that Orion (the server not agent) needs SQLite >= 3.24.0 but Amazon Linux 2 only has 3.7.17 as it’s latest version.
Okay, I have downloaded and installed from source … now when I run 
sqlite3 --version
I get 
3.40.0 2022-11-16 12:10:08 89c459e766ea7e9165d0beeb124708b955a4950d0f4792f457465d71b158d318
$ /home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/bin/prefect agent start -q tenzinga_django_prod
Starting v2.7.0 agent with ephemeral API...
Traceback (most recent call last):
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 205, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 156, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/cli/agent.py", line 144, in start
    async with OrionAgent(
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/agent.py", line 573, in __aenter__
    await self.start()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/agent.py", line 554, in start
    await self.client.__aenter__()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/client/orion.py", line 1874, in __aenter__
    self._ephemeral_lifespan = await self._exit_stack.enter_async_context(
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/contextlib.py", line 619, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/client/base.py", line 90, in app_lifespan_context
    await context.__aenter__()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/asgi_lifespan/_manager.py", line 92, in __aenter__
    await self._exit_stack.aclose()
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/contextlib.py", line 656, in aclose
    await self.__aexit__(None, None, None)
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/contextlib.py", line 714, in __aexit__
    raise exc_details[1]
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/contextlib.py", line 697, in __aexit__
    cb_suppress = await cb(*exc_details)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/asgi_lifespan/_concurrency/asyncio.py", line 80, in __aexit__
    await self.task
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/asgi_lifespan/_concurrency/asyncio.py", line 63, in run_and_silence_cancelled
    await self.coroutine()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/asgi_lifespan/_manager.py", line 64, in run_app
    await <http://self.app|self.app>(scope, self.receive, self.send)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/fastapi/applications.py", line 270, in __call__
    await super().__call__(scope, receive, send)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/starlette/middleware/errors.py", line 149, in __call__
    await <http://self.app|self.app>(scope, receive, send)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/starlette/middleware/cors.py", line 76, in __call__
    await <http://self.app|self.app>(scope, receive, send)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 55, in __call__
    await <http://self.app|self.app>(scope, receive, send)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await <http://self.app|self.app>(scope, receive, send)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/starlette/routing.py", line 695, in __call__
    await self.lifespan(scope, receive, send)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/starlette/routing.py", line 671, in lifespan
    async with self.lifespan_context(app):
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/starlette/routing.py", line 648, in startup
    await handler()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/orion/api/server.py", line 345, in run_migrations
    await db.create_db()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/orion/database/interface.py", line 55, in create_db
    await self.run_migrations_upgrade()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/orion/database/interface.py", line 63, in run_migrations_upgrade
    await run_sync_in_worker_thread(alembic_upgrade)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 69, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/orion/database/alembic_commands.py", line 29, in alembic_upgrade
    alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/alembic/command.py", line 322, in upgrade
    script.run_env()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/orion/database/migrations/env.py", line 147, in <module>
    apply_migrations()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 201, in coroutine_wrapper
    return run_async_from_worker_thread(async_fn, *args, **kwargs)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 152, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/orion/database/migrations/env.py", line 137, in apply_migrations
    engine = await db_interface.engine()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/orion/database/interface.py", line 73, in engine
    engine = await self.database_config.engine()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/orion/database/configurations.py", line 189, in engine
    raise RuntimeError(
RuntimeError: Orion requires sqlite >= 3.24.0 but we found version 3.7.17
An exception occurred.
✅ 1
r

Ryan Peden

12/07/2022, 12:57 AM
Hmm. I wonder if the system Python was built linked to the system version of 
libsqlite3
, or something like that. On Amazon Linux and Centos/RHEL 7 (very similar to Amazon Linux, with a similarly-old sqlite3 version installed), the easiest past I've found to make this work is using Miniconda or Anaconda: https://discourse.prefect.io/t/how-to-run-prefect-2-on-centos-7-or-rhel-7/1507
j

Jarvis Stubblefield

12/07/2022, 12:58 AM
Ryan, I have the option of changing my OS… was trying to keep it the exact same as production for simplicity, but really getting tired of having to “work around” Amazon Linux
I did install/compile Python 3.10 before I did the SQLite install… maybe I should remove and install again…
What would be a recommended OS for when I get tired of messing with AL2
I use Pyenv to install/compile Python
r

Ryan Peden

12/07/2022, 1:04 AM
Ah - I think you could maybe re-build Python and keep an eye on the LD_LIBRARY_PATH when you do to make sure it uses your newer 
sqlite3
. I think you can set that when using 
pyenv
. There is a decent article on it; I'll see if I can find it. If you get tired of AL2, one of the LTS Ubuntu releases like 20.04 or 22.04 is usually pretty pain-free. The last couple Debian stable releases should be fine, too. I haven't kept up with things in the Fedora/RH/CentOS world as much, but one of the newer CentOS replacements like Rocky Linux should work without too much fuss (But I haven't tried it, so I can't speak from experience)
I think this is what I had in mind: https://number1.co.za/upgrading-sqlite-on-centos-to-3-8-3-or-later/ It's about CentOS, but AL2 is based on RHEL just like CentOS, so I think everything in the article still applies
j

Jarvis Stubblefield

12/07/2022, 1:07 AM
Yes sir! I’ve been thinking about going Ubuntu, so I might use this as the excuse, but I’ll try to make it happen first hehe… thanks for the article and rapid responses!!
Okay now I have Python seeing the correct SQLite3 version … so lets see if we get a running agent! 🙂
r

Ryan Peden

12/07/2022, 1:39 AM
Sounds promising!
j

Jarvis Stubblefield

12/07/2022, 1:39 AM
Woohoo! @Ryan Peden Thanks for your assist!
It’s running! … now to ensure it runs just as good on Systemd as a final test, but it’s running with the Systemd command copy and pasted into the prompt … so I’m reasonably certain all is well! 🙂
r

Ryan Peden

12/07/2022, 1:40 AM
Happy to assist! Victory over old software versions is always satisfying 😄
j

Jarvis Stubblefield

12/07/2022, 1:40 AM
Yes it is!! 😄
So now I have an issue when trying to run the deployments from that machine so it registers with the appropriate environment values… 
$ python -m base.flows.deployments.maint_tasks
WARNING: Active profile 'tenzinga' set by environment variable not found. The default profile will be used instead.
Traceback (most recent call last):
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ec2-user/tenzinga/ppower-bknd/base/flows/deployments/maint_tasks.py", line 20, in <module>
    deployment = Deployment.build_from_flow(
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 205, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 156, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/home/ec2-user/.local/share/virtualenvs/ppower-bknd-UDByiVrD/lib/python3.10/site-packages/prefect/deployments.py", line 703, in build_from_flow
    entry_path = Path(flow_file).absolute().relative_to(Path(".").absolute())
  File "/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/pathlib.py", line 818, in relative_to
    raise ValueError("{!r} is not in the subpath of {!r}"
ValueError: '/home/ec2-user/tenzinga/ppower/base/flows/maint_tasks.py' is not in the subpath of '/home/ec2-user/tenzinga/ppower-bknd' OR one path is relative and the other is absolute.
This 
/home/ec2-user/tenzinga/ppower-bknd
is the actual directory … however 
/home/ec2-user/tenzinga/ppower
is a link to the first path.
For more context … this is the deployment file I’m using… 
# -*- coding: utf-8
from __future__ import unicode_literals

from prefect.deployments import Deployment
from prefect.orion.schemas.schedules import CronSchedule

from tenzinga_django_setup import prepare_django_and_paths
prepare_django_and_paths()

from ppower.base.flows.maint_tasks import run_maintenance_tasks


CRON_DAILY_AT_2_45_PM = "45 14 * * *"

deployment = Deployment.build_from_flow(
    flow=run_maintenance_tasks,
    name=f"daily_maintenance_tasks",
    description=f"Daily maintenance tasks for Tenzinga",
    tags=["maintenance", "daily"],
    schedule=CronSchedule(cron=CRON_DAILY_AT_2_45_PM),
    work_queue_name="tenzinga_django_prod",
)
deployment.apply()
This is the 
tenzinga_django_setup
file… 
# -*- coding: utf-8
from __future__ import unicode_literals
import os
import sys
import django
from pathlib import Path


def prepare_django_and_paths():
    # ensure we are in the path and are the working directory
    ppower_path = Path(os.environ.get("SRCDIR"))
    tenzinga_path = ppower_path.parent
    # Add paths to sys.path if not existing
    if str(tenzinga_path) not in sys.path:
        sys.path.append(str(tenzinga_path))
    if str(ppower_path) not in sys.path:
        sys.path.append(str(ppower_path))

    # set the default Django settings module for this module.
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "ppower.settings")
    django.setup()

    return tenzinga_path, ppower_path
I have tried changing 
SRCDIR
environment variable between both 
/home/ec2-user/tenzinga/ppower
and `/home/ec2-user/tenzinga/ppower-bknd`to no avail so I returned it back to the 
ppower
version.
I potentially could put the directory 
ppower-bknd
back to 
ppower
and remove the link of course before doing so. I just don’t recall why I set it up that way in the first place. I think it had to do with the original 
ppower
directory was a Mercurial repo and before getting it fully moved over I didn’t want to delete it or something…
I’ll try that and see if anything blows up in my face. 🙂
Now I know why I didn’t move it … 
pipenv
changes for it as it sees it as a different directory…
Well that was at least one hindrance that was added lol
So I would need to edit my systemd services to give them the new path once I have it working…
Okay … that seems to have fixed it up … I have a deployment set in Prefect Cloud … going to check it out.
Might be a good thing I had to do that… been wanting it back in the 
ppower
directory for a while. 🙂
k

Kalise Richmond

12/07/2022, 3:43 AM
Hey @Jarvis Stubblefield in the future can you put code and tracebacks in the related threads to help keep community channel organized?
👍 1
j

Jarvis Stubblefield

12/07/2022, 3:56 AM
Yes ma’am. I usually do… was just going fast this time and wasn’t originally going to post it. Will ensure I do it in the future!
k

Kalise Richmond

12/07/2022, 4:06 AM
Thanks! We all do it sometimes 😉
It looks like you managed to solve your problem here 😄 Are you running into any other issues?
j

Jarvis Stubblefield

12/07/2022, 7:35 AM
We’ll see tomorrow. 🙂
It’ll officially be running on production tasks.
@Kalise Richmond Well … sadly I have an issue.
I probably should have asked last night… posting a new thread.
2 Views
#prefect-communityJoin Slack
Powered by