Matt Camp

    Matt Camp

    1 year ago
    I'm trying to setup an agent for my local Prefect Server and pretty sure I did something wrong.. The agent registers itself, but the UI says that the Agent hasn't checked for any flows. I think that it might have something to do with how I've configured the Prefect server. It's running in a docker stack with everything being routed through Traefik. This is my agent's config.yml :
    # debug mode
    debug = true
    
    # base configuration directory (typically you won't change this!)
    home_dir = "~/.prefect/"
    
    backend = "server"
    
    [server]
    host = "<http://apollo.prefect.campground.lan>"
    port = "4200"
    endpoint = "${server.host}:${server.port}"
    my apollo server is at
    <http://apollo.prefect.campground.lan:4200/>
    my graphl server is at
    <http://graphql.prefect.campground.lan:4200/graphql>
    my ui is at
    <http://prefect.campground.lan>
    Any glaring problems with what I've done? TIA
    Michael Adkins

    Michael Adkins

    1 year ago
    Hey @Matt Camp -- no glaring problems here. Can you do
    PREFECT__CLOUD__AGENT__LEVEL=DEBUG prefect agent local start
    (or however you start your agents) and share a bit of the logs?
    Matt Camp

    Matt Camp

    1 year ago
    thanks! I just ran that and it looks like my agent isn't able to load some of the dependencies of one of the flows maybe?
    myagent  | [2021-05-17 20:24:20,734] DEBUG - CampNetAgent | Completed flow run submission (id: 073c2eef-abee-4fb3-8292-7b36d841829a)
    myagent  | [2021-05-17 20:24:20,735] DEBUG - CampNetAgent | Completed flow run submission (id: c7396f57-60e0-42e6-b129-d21b49253e34)
    myagent  | [2021-05-17 20:24:20,743] DEBUG - CampNetAgent | Completed flow run submission (id: 672206dd-76e5-472a-a3b8-0a064a01bf05)
    myagent  | [2021-05-17 20:24:21,030] DEBUG - CampNetAgent | Querying for flow runs
    myagent  | [2021-05-17 20:24:21,055] DEBUG - CampNetAgent | No flow runs found
    myagent  | [2021-05-17 20:24:21,056] DEBUG - CampNetAgent | Next query for flow runs in 1.0 seconds
    myagent  | No module named 'weather'
    myagent  | Traceback (most recent call last):
    myagent  |   File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    myagent  |     return _run_code(code, main_globals, None,
    myagent  |   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    myagent  |     exec(code, run_globals)
    myagent  |   File "/usr/local/lib/python3.9/site-packages/prefect/__main__.py", line 4, in <module>
    myagent  |     cli()
    myagent  |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1134, in __call__
    myagent  |     return self.main(*args, **kwargs)
    myagent  |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1059, in main
    myagent  |     rv = self.invoke(ctx)
    myagent  |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1665, in invoke
    myagent  |     return _process_result(sub_ctx.command.invoke(sub_ctx))
    myagent  |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1665, in invoke
    myagent  |     return _process_result(sub_ctx.command.invoke(sub_ctx))
    myagent  |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1401, in invoke
    myagent  |     return ctx.invoke(self.callback, **ctx.params)
    myagent  |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 767, in invoke
    myagent  |     return __callback(*args, **kwargs)
    myagent  |   File "/usr/local/lib/python3.9/site-packages/prefect/cli/execute.py", line 90, in flow_run
    myagent  |     raise exc
    myagent  |   File "/usr/local/lib/python3.9/site-packages/prefect/cli/execute.py", line 67, in flow_run
    myagent  |     flow = storage.get_flow(flow_data.name)
    myagent  |   File "/usr/local/lib/python3.9/site-packages/prefect/storage/local.py", line 102, in get_flow
    myagent  |     return extract_flow_from_module(
    myagent  |   File "/usr/local/lib/python3.9/site-packages/prefect/utilities/storage.py", line 124, in extract_flow_from_module
    myagent  |     module = importlib.import_module(mod_name)
    myagent  |   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    myagent  |     return _bootstrap._gcd_import(name[level:], package, level)
    myagent  |   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
    myagent  |   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
    myagent  |   File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
    myagent  |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
    myagent  |   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
    myagent  |   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
    myagent  |   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
    myagent  | ModuleNotFoundError: No module named 'weather'
    myagent  | No module named 'weather'
    does my Agent need to have access to any of the folders on the server?
    Michael Adkins

    Michael Adkins

    1 year ago
    So it looks like you're using
    Local
    storage here? If you're using local storage then your agent needs access to the files. You could use remote storage instead like
    S3
    or
    GitHub
    storage so that you don't need the flow source on the agent.
    Matt Camp

    Matt Camp

    1 year ago
    ahh ok yea I've just got the Agent in Docker container and then the Sever is in its own stack of containers.. hmm so do I need to make
    ~/.prefect
    accessible as a volume to everything or is it a different folder?
    Michael Adkins

    Michael Adkins

    1 year ago
    Local
    storage defaults to
    ~/.prefect/flows
    but you can make it whatever you'd like
    Mounting it as you've described should work
    Matt Camp

    Matt Camp

    1 year ago
    alright i'll give that a try.. thank you!
    ok I'm still doing something wrong. I have a folder
    $PWD/prefect/prefect
    that I am mapping to the Graphql container and my Agent at
    /root/.prefect
    and I'm still having issues. my Agent says
    ____            __           _        _                    _
    |  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
    | |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
    |  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
    |_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                               |___/
    
    [2021-05-18 01:43:38,319] INFO - agent | Starting LocalAgent with labels ['0f43a1ca6a0f']
    [2021-05-18 01:43:38,319] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
    [2021-05-18 01:43:38,319] INFO - agent | Agent connecting to the Prefect API at <http://apollo.prefect.campground.lan:4200>
    [2021-05-18 01:43:38,325] INFO - agent | Waiting for flow runs...
    [2021-05-18 01:43:38,325] DEBUG - agent | Running agent heartbeat...
    [2021-05-18 01:43:38,328] DEBUG - agent | Max Workers: 12
    [2021-05-18 01:43:38,329] DEBUG - agent | Sleeping heartbeat for 60.0 seconds
    [2021-05-18 01:43:38,329] DEBUG - agent | Querying for flow runs
    [2021-05-18 01:43:38,346] DEBUG - agent | No flow runs found
    [2021-05-18 01:43:38,347] DEBUG - agent | Next query for flow runs in 0.5 seconds
    [2021-05-18 01:43:38,847] DEBUG - agent | Querying for flow runs
    [2021-05-18 01:43:38,866] DEBUG - agent | No flow runs found
    [2021-05-18 01:43:38,866] DEBUG - agent | Next query for flow runs in 1.0 seconds
    [2021-05-18 01:43:39,866] DEBUG - agent | Querying for flow runs
    [2021-05-18 01:43:39,885] DEBUG - agent | No flow runs found
    [2021-05-18 01:43:39,885] DEBUG - agent | Next query for flow runs in 2.0 seconds
    [2021-05-18 01:43:41,885] DEBUG - agent | Querying for flow runs
    [2021-05-18 01:43:41,951] DEBUG - agent | No flow runs found
    [2021-05-18 01:43:41,952] DEBUG - agent | Next query for flow runs in 4.0 seconds
    The UI says its unhealthy
    Michael Adkins

    Michael Adkins

    1 year ago
    Starting LocalAgent with labels ['0f43a1ca6a0f']
    does your flow run have a matching label? You may want to start the agent with --no-hostname-label
    Matt Camp

    Matt Camp

    1 year ago
    ok I can't figure this out.. I have a docker stack that runs Prefect with Traefik and some other thing. I go to create an Agent the Agent is "unhealthy" as discussed above. BUT I have taken the EXACT same stack (copied the entire directory).. to another computer on my network and brought it up. Agents work just fine. So I've decided my config is fine, its something with my system config. Any suggestions on things I could check?
    Michael Adkins

    Michael Adkins

    1 year ago
    The
    prefect diagnostics
    command should show all config changes
    Matt Camp

    Matt Camp

    1 year ago
    FINALLY! figured it out! my system time was off by 13-14 minutes.. figured it out by staring at the unhealthy message on the UI constantly alternate between saying the last time the Agent queried for a Flow was 13-14 minutes ago. Thanks for the help!