https://prefect.io logo
Channels
Powered by
c

Claire Herdeman

11/10/2022, 2:50 PM
Hey! I'm working through this template to get an agents up and running as ECS Service. I've got it running and the agent successfully created the work queue in our Prefect Cloud account. However, the agent logs say that the agent is checking for flow runs, but the work queue in Prefect Cloud is "Unhealthy" and says that the queue has never been polled. Could you help me understand how work queues send and receive updates from the agent? Since the command to create the queue was able to get out, I'm guessing that the issue has to do with our security infrastructure and our rules for inbound traffic. Understanding the communication would help us debug. Thank you!
c

Christopher Boyd

11/10/2022, 2:59 PM
Hi Claire, Agents are outbound connections only
if you are running the agent in ECS on a vm or something, you could log into the system and test a curl connection to the api to verify your outbound rules are allowing outbound calls
Copy code
$ curl -vvvv <https://api.prefect.cloud>
*   Trying 34.111.36.90:443...
* Connected to api.prefect.cloud (34.111.36.90) port 443 (#0)
c

Claire Herdeman

11/10/2022, 3:14 PM
Thanks that's helpful! Will report back
🙌 1
I've been taking a look at this off an on, this is running as a task so its a bit more difficult to get into the container directly. I looked at the ENI associated with the task and saw that traffic was getting rejected, made some tweaks and was able to resolve. The agent still doesn't seem to be connecting though, it doesn't create the queue on startup and doesn't poll. Does anything look off to you about this startup message? does the URL look good, etc?
Copy code
2022-11-10 17:05:50 Agent started! Looking for work from queue(s): dev...
2022-11-10 17:05:50 23:05:50.200 | DEBUG | prefect.client - Connecting to API at
2022-11-10 17:05:50 %7B%22PREFECT_API_KEY%22:%22<key>%22,%22PREFE
2022-11-10 17:05:50 CT_API_URL%22:%22<https://api.prefect.cloud/api/accounts/><acct_id>
2022-11-10 17:05:50 /workspaces/<ws_id>/
2022-11-10 17:05:50 Starting v2.6.7 agent connected to
2022-11-10 17:05:50 {"PREFECT_API_KEY":"<key>","PREFECT_API_URL":
2022-11-10 17:05:50 "<https://api.prefect.cloud/api/accounts/><acct_id>/wor
2022-11-10 17:05:50 kspaces/<ws_id>"}...
Welll now my suspicion is that its just how the secret is getting passed, looks like it should just say the url there. Though it still says that it connects which seems odd?
c

Christopher Boyd

11/11/2022, 1:12 PM
it looks like you have a carriage return in your deployment specification
Copy code
2022-11-10 17:05:50 %7B%22PREFECT_API_KEY%22:%22<key>%22,%22PREFE
2022-11-10 17:05:50 CT_API_URL%22:%22<https://api.prefect.cloud/api/accounts/><acct_id>
let me double check on my end real quick with debug logs
I think your deployment spect might have some awry wiht your API key; it looks like this for me; there is no reference to my API key in logs
Copy code
13:15:09.004 | DEBUG   | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/e><acct_id>/workspaces/<ws_id/
for context my spec:
Copy code
env:
            - name: PREFECT_API_URL
              value: >-
                <https://api.prefect.cloud/api/accounts/><+++>/workspaces/<+++>
            - name: PREFECT_API_KEY
              value: <+++>
            - name: PREFECT_LOGGING_LEVEL
              value: DEBUG
c

Claire Herdeman

11/11/2022, 3:01 PM
Perfect, yeah i think there's something wonky with how the secrets are getting pulled, though I'm confused why it would say that the agent has started though if those items aren't reading?
c

Christopher Boyd

11/11/2022, 3:27 PM
The agent starting as a service is successful, just connecting to the API is not; you could put www.google.com as your API_URL, and the service will still say that it started successfully, but not pull proper data. this is due to teh fact that every consumers API URL would presumably be different, so it has to check and return a result
c

Claire Herdeman

11/11/2022, 3:48 PM
Gochya, we're up and running now! I do think this should have thrown an error though. Basically I wasn't pulling the values out of the json blob of the secret, so the full json object was going into the slot for both API url and the API key. Both fields would have been totally malformed, like even if google.com would it seems like it should recognize if its NOT a url or if the url is in the incorrect domain.
🙌 1
c

Christopher Boyd

11/11/2022, 4:10 PM
That’s good feedback and I can raise that with the team
👍 1
6 Views
Powered by