Hey Team! I see that the nomad-agent was deprecate...
# prefect-serverc
Charles Leung
10/01/2020, 5:43 PMHey Team! I see that the nomad-agent was deprecated; is there a particular reason this was the case?
c
Chris White
10/01/2020, 6:36 PM
Hi Charles! We originally built that for an early customer that no longer uses nomad, and were finding it was a lot of work to continue maintaining when no one was using it. Definitely open to resurrecting it if there’s community interest - maybe open a GitHub issue if you’d like to see that happen
c
Charles Leung
10/01/2020, 6:38 PMOk thank you chris! that's helpful insight -
👍 1
Charles Leung
10/01/2020, 8:57 PMHey chris - following up on this; my team still uses nomad but i believe on the longer term we are switching to using k8s. however i tried to just run nomad agent to see how it fares right now using prefect==0.9.8 (the last one before max_poll seems to not make it work) - it seems it's not picking up my tasks; does it work at its current state or was it just a POC?
c
Chris White
10/01/2020, 8:58 PM
Hmmmm I definitely can’t guarantee it will work, but I would expect it to work. Does it have labels matching those of your Flows?
c
Charles Leung
10/01/2020, 8:58 PMyep! both have matching labels
c
Chris White
10/01/2020, 8:59 PM
are you running against Prefect Server or Prefect Cloud?
c
Charles Leung
10/01/2020, 9:00 PMprefect cloud
c
Chris White
10/01/2020, 9:02 PM
hmmm yea 0.9.8 is compatible with Cloud for sure… are you seeing any logs from the agent?
c
Charles Leung
10/01/2020, 9:03 PMFound flow runs ['ac588a53-f1a5-40bf-a4d0-3d02250a3b12']
[2020-10-01 210311,458] DEBUG - agent | Querying flow run metadata
[2020-10-01 210311,765] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-10-01 210311,766] DEBUG - agent | Updating states for flow run ac588a53-f1a5-40bf-a4d0-3d02250a3b12
[2020-10-01 210311,766] DEBUG - agent | Next query for flow runs in 0.25 seconds
[2020-10-01 210311,767] DEBUG - agent | Flow run ac588a53-f1a5-40bf-a4d0-3d02250a3b12 is in a Scheduled state, updating to Submitted
[2020-10-01 210312,018] DEBUG - agent | Querying for flow runs
[2020-10-01 210312,333] DEBUG - agent | No flow runs found
[2020-10-01 210312,333] DEBUG - agent | Next query for flow runs in 0.5 seconds
[2020-10-01 210312,386] DEBUG - agent | Completed flow run submission (id: ac588a53-f1a5-40bf-a4d0-3d02250a3b12)
Charles Leung
10/01/2020, 9:04 PMpretty much the job is stuck at the submission state -
c
Chris White
10/01/2020, 9:06 PM
gotcha, then that is most likely an issue with how the agent is talking to nomad which might be hard to debug since it’s such an old version of Prefect. If you’re up for the challenge, it could be really useful to dig into the individual steps that the agent is taking when it creates the nomad job and see where it’s failing — it seems that the nomad client itself isn’t erroring out
c
Charles Leung
10/01/2020, 9:08 PMgotcha thanks 🙂. I'll review it when i have the chance!
c
Chris White
10/01/2020, 9:09 PM
sounds good! if you learn anything, that could be a really useful first step in re-implementing the nomad agent
2 Views