I'm trying to setup an agent for my local Prefect ...
# prefect-server
m
I'm trying to setup an agent for my local Prefect Server and pretty sure I did something wrong.. The agent registers itself, but the UI says that the Agent hasn't checked for any flows. I think that it might have something to do with how I've configured the Prefect server. It's running in a docker stack with everything being routed through Traefik. This is my agent's config.yml :
Copy code
# debug mode
debug = true

# base configuration directory (typically you won't change this!)
home_dir = "~/.prefect/"

backend = "server"

[server]
host = "<http://apollo.prefect.campground.lan>"
port = "4200"
endpoint = "${server.host}:${server.port}"
my apollo server is at
<http://apollo.prefect.campground.lan:4200/>
my graphl server is at
<http://graphql.prefect.campground.lan:4200/graphql>
my ui is at
<http://prefect.campground.lan>
Any glaring problems with what I've done? TIA
z
Hey @Matt Camp -- no glaring problems here. Can you do
PREFECT__CLOUD__AGENT__LEVEL=DEBUG prefect agent local start
(or however you start your agents) and share a bit of the logs?
m
thanks! I just ran that and it looks like my agent isn't able to load some of the dependencies of one of the flows maybe?
Copy code
myagent  | [2021-05-17 20:24:20,734] DEBUG - CampNetAgent | Completed flow run submission (id: 073c2eef-abee-4fb3-8292-7b36d841829a)
myagent  | [2021-05-17 20:24:20,735] DEBUG - CampNetAgent | Completed flow run submission (id: c7396f57-60e0-42e6-b129-d21b49253e34)
myagent  | [2021-05-17 20:24:20,743] DEBUG - CampNetAgent | Completed flow run submission (id: 672206dd-76e5-472a-a3b8-0a064a01bf05)
myagent  | [2021-05-17 20:24:21,030] DEBUG - CampNetAgent | Querying for flow runs
myagent  | [2021-05-17 20:24:21,055] DEBUG - CampNetAgent | No flow runs found
myagent  | [2021-05-17 20:24:21,056] DEBUG - CampNetAgent | Next query for flow runs in 1.0 seconds
myagent  | No module named 'weather'
myagent  | Traceback (most recent call last):
myagent  |   File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
myagent  |     return _run_code(code, main_globals, None,
myagent  |   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
myagent  |     exec(code, run_globals)
myagent  |   File "/usr/local/lib/python3.9/site-packages/prefect/__main__.py", line 4, in <module>
myagent  |     cli()
myagent  |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1134, in __call__
myagent  |     return self.main(*args, **kwargs)
myagent  |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1059, in main
myagent  |     rv = self.invoke(ctx)
myagent  |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1665, in invoke
myagent  |     return _process_result(sub_ctx.command.invoke(sub_ctx))
myagent  |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1665, in invoke
myagent  |     return _process_result(sub_ctx.command.invoke(sub_ctx))
myagent  |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1401, in invoke
myagent  |     return ctx.invoke(self.callback, **ctx.params)
myagent  |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 767, in invoke
myagent  |     return __callback(*args, **kwargs)
myagent  |   File "/usr/local/lib/python3.9/site-packages/prefect/cli/execute.py", line 90, in flow_run
myagent  |     raise exc
myagent  |   File "/usr/local/lib/python3.9/site-packages/prefect/cli/execute.py", line 67, in flow_run
myagent  |     flow = storage.get_flow(flow_data.name)
myagent  |   File "/usr/local/lib/python3.9/site-packages/prefect/storage/local.py", line 102, in get_flow
myagent  |     return extract_flow_from_module(
myagent  |   File "/usr/local/lib/python3.9/site-packages/prefect/utilities/storage.py", line 124, in extract_flow_from_module
myagent  |     module = importlib.import_module(mod_name)
myagent  |   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
myagent  |     return _bootstrap._gcd_import(name[level:], package, level)
myagent  |   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
myagent  |   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
myagent  |   File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
myagent  |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
myagent  |   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
myagent  |   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
myagent  |   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
myagent  | ModuleNotFoundError: No module named 'weather'
myagent  | No module named 'weather'
does my Agent need to have access to any of the folders on the server?
z
So it looks like you're using
Local
storage here? If you're using local storage then your agent needs access to the files. You could use remote storage instead like
S3
or
GitHub
storage so that you don't need the flow source on the agent.
m
ahh ok yea I've just got the Agent in Docker container and then the Sever is in its own stack of containers.. hmm so do I need to make
~/.prefect
accessible as a volume to everything or is it a different folder?
z
Local
storage defaults to
~/.prefect/flows
but you can make it whatever you'd like
Mounting it as you've described should work
m
alright i'll give that a try.. thank you!
ok I'm still doing something wrong. I have a folder
$PWD/prefect/prefect
that I am mapping to the Graphql container and my Agent at
/root/.prefect
and I'm still having issues. my Agent says
Copy code
____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2021-05-18 01:43:38,319] INFO - agent | Starting LocalAgent with labels ['0f43a1ca6a0f']
[2021-05-18 01:43:38,319] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2021-05-18 01:43:38,319] INFO - agent | Agent connecting to the Prefect API at <http://apollo.prefect.campground.lan:4200>
[2021-05-18 01:43:38,325] INFO - agent | Waiting for flow runs...
[2021-05-18 01:43:38,325] DEBUG - agent | Running agent heartbeat...
[2021-05-18 01:43:38,328] DEBUG - agent | Max Workers: 12
[2021-05-18 01:43:38,329] DEBUG - agent | Sleeping heartbeat for 60.0 seconds
[2021-05-18 01:43:38,329] DEBUG - agent | Querying for flow runs
[2021-05-18 01:43:38,346] DEBUG - agent | No flow runs found
[2021-05-18 01:43:38,347] DEBUG - agent | Next query for flow runs in 0.5 seconds
[2021-05-18 01:43:38,847] DEBUG - agent | Querying for flow runs
[2021-05-18 01:43:38,866] DEBUG - agent | No flow runs found
[2021-05-18 01:43:38,866] DEBUG - agent | Next query for flow runs in 1.0 seconds
[2021-05-18 01:43:39,866] DEBUG - agent | Querying for flow runs
[2021-05-18 01:43:39,885] DEBUG - agent | No flow runs found
[2021-05-18 01:43:39,885] DEBUG - agent | Next query for flow runs in 2.0 seconds
[2021-05-18 01:43:41,885] DEBUG - agent | Querying for flow runs
[2021-05-18 01:43:41,951] DEBUG - agent | No flow runs found
[2021-05-18 01:43:41,952] DEBUG - agent | Next query for flow runs in 4.0 seconds
The UI says its unhealthy
z
Starting LocalAgent with labels ['0f43a1ca6a0f']
does your flow run have a matching label? You may want to start the agent with` --no-hostname-label`
m
ok I can't figure this out.. I have a docker stack that runs Prefect with Traefik and some other thing. I go to create an Agent the Agent is "unhealthy" as discussed above. BUT I have taken the EXACT same stack (copied the entire directory).. to another computer on my network and brought it up. Agents work just fine. So I've decided my config is fine, its something with my system config. Any suggestions on things I could check?
z
The
prefect diagnostics
command should show all config changes
m
FINALLY! figured it out! my system time was off by 13-14 minutes.. figured it out by staring at the unhealthy message on the UI constantly alternate between saying the last time the Agent queried for a Flow was 13-14 minutes ago. Thanks for the help!