Hello everyone I m having trouble getting the `prefect agent

Question

Hello everyone I m having trouble getting the `prefect agent` to execute within my Django environment using Django ORM in the flow I finally got the deployment created by adding my project to the sys path before running the Django setup That works for running the flow manually but now I m getting another error when the Agent tries to run the flow Full error in the thread message So my idea at this point is to create a Django Management command which executes in the Django environment to run the agent through Python code I have an example of doing this in Prefect v1 but I m using Prefect v2 and haven t see where this is clearly documented `LocalAgent` no longer seems to exist in Prefect v2 to run an agent through code Any ideas of other ways to run this or how to run the agent from within Python would be super helpful and wholly appreciated

Jarvis Stubblefield · Answer

Here s the full agent run + error message output ``` $ lt git feature prefect tasks gt prefect agent start q tenzinga django dev Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Starting v2 4 1 agent connected to <https api prefect cloud api accounts 5626ffe9 0140 4e88 babc 4a4fc614bb99 workspaces ee8a533d 2754 420e 87f2 2d6b084984af> | | | | | | | | | | | || || | | | | || ` | | | | | | | | | | | | | | | | | | | Agent started Looking for work from queue s tenzinga django dev 14 25 57 159 | INFO | prefect agent Submitting flow run 92c51db6 de80 4b48 9abc 8c5228e03658 14 25 57 399 | INFO | prefect infrastructure process Opening process mahogany wildebeest 14 25 57 408 | INFO | prefect agent Completed submission of flow run 92c51db6 de80 4b48 9abc 8c5228e03658 Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve 14 26 26 401 | ERROR | Flow run mahogany wildebeest Flow could not be retrieved from deployment Traceback most recent call last File lt frozen importlib bootstrap external gt line 843 in exec module File lt frozen importlib bootstrap gt line 219 in call with frames removed File base flows log entry py line 22 in lt module gt django setup File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages django init py line 19 in setup configure logging settings LOGGING CONFIG settings LOGGING File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages django conf init py line 79 in getattr self setup name File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages django conf init py line 66 in setup self wrapped = Settings settings module File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages django conf init py line 157 in init mod = importlib import module self SETTINGS MODULE File Users ballisticpain pyenv versions anaconda3 2020 11 lib python3 8 importlib init py line 127 in import module return bootstrap gcd import name level package level File lt frozen importlib bootstrap gt line 1014 in gcd import File lt frozen importlib bootstrap gt line 991 in find and load File lt frozen importlib bootstrap gt line 961 in find and load unlocked File lt frozen importlib bootstrap gt line 219 in call with frames removed File lt frozen importlib bootstrap gt line 1014 in gcd import File lt frozen importlib bootstrap gt line 991 in find and load File lt frozen importlib bootstrap gt line 973 in find and load unlocked ModuleNotFoundError No module named ppower The above exception was the direct cause of the following exception Traceback most recent call last File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect engine py line 257 in retrieve flow then begin flow run flow = await load flow from flow run flow run client=client File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect client orion py line 80 in with injected client return await fn args kwargs File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect deployments py line 70 in load flow from flow run flow = await run sync in worker thread import object str import path File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect utilities asyncutils py line 57 in run sync in worker thread return await anyio to thread run sync call cancellable=True File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages anyio backends asyncio py line 867 in run result = context run func args File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect utilities importtools py line 193 in import object module = load script as module script path File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect utilities importtools py line 156 in load script as module raise ScriptError user exc=exc path=path from exc prefect exceptions ScriptError Script at base flows log entry py encountered an exception 14 26 32 275 | INFO | prefect infrastructure process Process mahogany wildebeest exited cleanly ```

Jarvis Stubblefield · Answer

So I have found the `OrionAgent` and have attempted to run a Django Command using that however that s using `async await` and I have zero experience there Here s what I am trying so far

Jarvis Stubblefield · Answer

``` coding utf 8 from future import unicode literals Custom Django management command to work with Prefect Via various flags you can run a specific flow on demand register flows with the server or run a local agent import os from typing import Any from django core management base import BaseCommand CommandError CommandParser from prefect deployments import LocalFileSystem Deployment from prefect agent import OrionAgent from deployments flows reset timeseries end times import flow as reset timeseries end times from deployments flows import default dataset refresh flows = reset timeseries end times reset timeseries end times default dataset refresh flow name default dataset refresh flow daily = Schedule clocks= CronClock 5 6 class Command BaseCommand help = Run Prefect flow registration agent or a single flow def add arguments self parser CommandParser gt None parser add argument agent help= Run local Prefect Agent action= store true parser add argument run flow help= Run a specified flow by name def handle self args Any options Any if options run flow try flow = flows options run flow except KeyError raise CommandError f Specified flow options run flow does not exist flow run if options agent self agent else self stdout write Not running local Prefect Agent async def agent self Run a local Prefect agent self stdout write Running local Prefect Agent agent = OrionAgent tenzinga django dev await agent start ```

Jarvis Stubblefield · Answer

As you can see I ve started adding `async await` to the various places based on the error output but I don t know how to turn a Django command into one that works with async await

Ryan Peden · Answer

I m not sure if running an Agent that way will help The agent is running the script in a separate process so it won t see anything you added to sys path and can t find the ppower module Is your goal here to use the ORM models from an existing Django app inside your flow

Jarvis Stubblefield · Answer

Ryan thanks for responding yes my intention is to use Prefect to run several background tasks and or scheduled tasks Some may be triggered by an action on our app and others are just the scheduled tasks

Jarvis Stubblefield · Answer

The Django ORM being the biggest bit that needs Django

Jarvis Stubblefield · Answer

I was hoping that it would grab the environment copy from the parents when it starts the new process

Jarvis Stubblefield · Answer

If that isn t the correct way then how should I go about it

Ryan Peden · Answer

Actually looking through the entire exception I might be incorrect about what s going wrong here Is there any chance you could share the part of your code that is adding the project to the sys path

Jarvis Stubblefield · Answer

< Ryan Peden> Certainly let me throw that in here and show you

Jarvis Stubblefield · Answer

My Django is a little odd in it s structure which has always made pathing stuff a bit difficult usually I find a way though

Jarvis Stubblefield · Answer

Here s the top of my `log entry py` file The rest of the code are the tasks and flows I can show them to you if you feel they are useful but I think this bit at the top is the relevant bit ``` coding utf 8 from future import unicode literals import casefy import csv import datetime import django import os from pathlib import Path import sys from dateutil relativedelta import relativedelta from django db models import Q from prefect import flow task get run logger from typing import List Dict set the default Django settings module for this module os environ setdefault DJANGO SETTINGS MODULE ppower settings ppower path = Path cwd parent sys path append str ppower path absolute django setup from ppower base models import Organization```

Jarvis Stubblefield · Answer

The path that `ppower path` returns during the `prefect build` is ` user stuff Development tenzinga` where `ppower` is a folder within that directory That s my local setup On production it is similar but obviously slightly different

Ryan Peden · Answer

Ok makes sense Since the agent is running your flow code in a subprocess that uses a different working directory it looks like perhaps the `Path cwd parent` call is adding the wrong directory to the sys path when the agent runs it

Jarvis Stubblefield · Answer

< Ryan Peden> I hadn t thought of that

Jarvis Stubblefield · Answer

I was originally trying to go from the ` file ` I wasn t able to get it to build until I used the `cwd` bit on `Path`

Jarvis Stubblefield · Answer

Trying a rebuild to see what other path options I have I just recalled after discussing this with you that I set a BASE PATH setting in the settings of Django I m doing that with `str Path file resolve parent ` I would need a couple of extra parent calls but testing now to see what I get

Ryan Peden · Answer

If you expect your Django code to always be in the same directory in production putting the absolute directory in an environment variable would also be an easy way to make sure any subprocess the agent starts can find it easily I m a Django fan myself so I m going to experiment more with Django Prefect integration Ideally I d like to create a collection of helpers that make it easy to use them together

Jarvis Stubblefield · Answer

I may already have an environment variable in production didn t think of that either slightly smiling face

Jarvis Stubblefield · Answer

I would be happy to help in anyway I can with the Django Prefect integration

Jarvis Stubblefield · Answer

I think it would greatly assist with adoption of Prefect within the Django community if it was easier to replace Celery with it

Jarvis Stubblefield · Answer

So that still failed even though I got the build to work I had it print the path this second time checking now if it is correct

Jarvis Stubblefield · Answer

Well I can t seem to see the output here s the error Same as before from what I can tell ```14 31 16 782 | INFO | prefect agent Submitting flow run cb62d36d 860a 4ec2 9445 6f0968ade880 14 31 17 275 | INFO | prefect infrastructure process Opening process omega6 oxkintok ring 14 31 17 284 | INFO | prefect agent Completed submission of flow run cb62d36d 860a 4ec2 9445 6f0968ade880 Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve Failed to parse annotation from Name node NoneType object has no attribute resolve 14 31 32 870 | ERROR | Flow run omega6 oxkintok ring Flow could not be retrieved from deployment Traceback most recent call last File lt frozen importlib bootstrap external gt line 843 in exec module File lt frozen importlib bootstrap gt line 219 in call with frames removed File base flows log entry py line 20 in lt module gt logger = get run logger File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect logging loggers py line 109 in get run logger raise RuntimeError There is no active flow or task run context RuntimeError There is no active flow or task run context The above exception was the direct cause of the following exception Traceback most recent call last File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect engine py line 257 in retrieve flow then begin flow run flow = await load flow from flow run flow run client=client File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect client orion py line 80 in with injected client return await fn args kwargs File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect deployments py line 70 in load flow from flow run flow = await run sync in worker thread import object str import path File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect utilities asyncutils py line 57 in run sync in worker thread return await anyio to thread run sync call cancellable=True File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages anyio backends asyncio py line 867 in run result = context run func args File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect utilities importtools py line 193 in import object module = load script as module script path File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect utilities importtools py line 156 in load script as module raise ScriptError user exc=exc path=path from exc prefect exceptions ScriptError Script at base flows log entry py encountered an exception 14 31 36 891 | INFO | prefect infrastructure process Process omega6 oxkintok ring exited cleanly ```

Jarvis Stubblefield · Answer

Also here s that relevant section of the `log entry` flow file ``` set the default Django settings module for this module os environ setdefault DJANGO SETTINGS MODULE ppower settings ppower path = str Path file resolve parent parent parent parent logger = get run logger <http logger info|logger info> ppower path print ppower path sys path append ppower path django setup from ppower base models import Organization```

Jarvis Stubblefield · Answer

I didn t see the ` info ` or `print ` output in the tasks results I pasted above

Jarvis Stubblefield · Answer

I recall seeing somewhere I can have the flow run re direct stdout print to the log

Jarvis Stubblefield · Answer

I don t recall where I would set that up

Jarvis Stubblefield · Answer

< Ryan Peden> I have gotten a bit further It is definitely a pathing issue with the agent process Here s the code bit ```try ensure we are in the path and are the working directory ppower path = str Path file resolve parent parent parent parent os chdir ppower path sys path append ppower path set the default Django settings module for this module os environ setdefault DJANGO SETTINGS MODULE ppower settings django setup except print Path ppower path ```

Jarvis Stubblefield · Answer

Here s the error output I added the to the Path after this run ```19 25 35 497 | INFO | prefect agent Submitting flow run 32f36e7e a51a 42d2 9138 a320330ffad4 19 25 35 650 | INFO | prefect infrastructure process Opening process omicron941 hupyria 19 25 35 657 | INFO | prefect agent Completed submission of flow run 32f36e7e a51a 42d2 9138 a320330ffad4 19 25 51 106 | ERROR | Flow run omicron941 hupyria Flow could not be retrieved from deployment Traceback most recent call last File lt frozen importlib bootstrap external gt line 843 in exec module File lt frozen importlib bootstrap gt line 219 in call with frames removed File base flows log entry py line 31 in lt module gt from ppower base models import Organization ModuleNotFoundError No module named ppower The above exception was the direct cause of the following exception Traceback most recent call last File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect engine py line 257 in retrieve flow then begin flow run flow = await load flow from flow run flow run client=client File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect client orion py line 82 in with injected client return await fn args kwargs File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect deployments py line 173 in load flow from flow run flow = await run sync in worker thread import object str import path File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect utilities asyncutils py line 68 in run sync in worker thread return await anyio to thread run sync call cancellable=True File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages anyio backends asyncio py line 867 in run result = context run func args File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect utilities importtools py line 193 in import object module = load script as module script path File Users ballisticpain local share virtualenvs ppower D20JDXzL lib python3 8 site packages prefect utilities importtools py line 156 in load script as module raise ScriptError user exc=exc path=path from exc prefect exceptions ScriptError Script at base flows log entry py encountered an exception Path private var folders sm c1qt38h11hj0psj9bgjg4vy80000gn T 19 25 55 121 | INFO | prefect infrastructure process Process omicron941 hupyria exited cleanly ```

Ryan Peden · Answer

Thanks for sharing that I m going to experiment with this in one of my Django apps over the weekend

Jarvis Stubblefield · Answer

< Ryan Peden> Feel free to reach out if you need anything or if I can be of assistance I appreciate your efforts on this

Jarvis Stubblefield · Answer

< Ryan Peden> I hope all is well with you and yours No worries if you didn t have any time over the weekend I know mine was busy If you have any further ideas even if not tested let me know I m still actively working to get this solved Thank you for your time and efforts to help