https://prefect.io logo
Join Slack
Channels
Powered by
Hi All, I am trying to launch my `prefect agent` ...
# ask-community
b
Hi All, I am trying to launch my 
prefect agent
in an ECS Fargate cluster. I am feeding in all my secrets via the cdk namely, 
RUNNER_TOKEN_FARGATE_AGENT
, 
AWS_SECRET_ACCESS_KEY
and 
AWS_ACCESS_KEY_ID
 I am getting some type of auth error when I run my image command
Copy code
CMD           [ "prefect", "agent", "ecs", "start", "--token", "$RUNNER_TOKEN_FARGATE_AGENT", \
                    "--task-role-arn=arn:aws:iam::********:role/ECSTaskS3Role", \
                    "--log-level", "INFO", "--label", "fargate-dev", "--label", "s3-flow-storage", \
                    "--name", "fargate-demo" ]
The trace in 🧵 `````` Any ideas would be great! I assumed I dont have to register the agent with Prefect cloud as I did this with Ec2 and it didnt have to be done. Thanks in advance for any help on this...
j
Hi @Ben Muller let me take a look at that for you. Would you mind moving the stack trace into the thread here and out of the main channel? (Makes it easier to read and to scroll through the channel more easliy)
n
Hi @Ben Muller, What type of token are you using for your fargate agent? Do you know if your token has an expiry date?
b
Copy code
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/agent/agent.py", line 890, in _setup_api_connection
self._verify_token(self.client.get_auth_token())
File "/usr/local/lib/python3.8/site-packages/prefect/agent/agent.py", line 824, in _verify_token
result = self.client.graphql(query="query { auth_info { api_token_scope } }")
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 311, in graphql
raise AuthorizationError(result["errors"])
prefect.utilities.exceptions.AuthorizationError: [{'path': ['auth_info'], 'message': 'AuthenticationError: Forbidden', 'extensions': {'code': 'UNAUTHENTICATED'}}]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/prefect", line 8, in <module>
done @Jenny
👍 1
Hi @Natalie Smith I am using a RUNNER token that I generated with the command 
prefect auth create-token -n DemoFargateAgentToken -s RUNNER
FYI this is what my Dockerfile looks like
Copy code
FROM          prefecthq/prefect:latest-python3.8

RUN           pip3 install "prefect[aws]" 

ENV           AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION
ENV           RUNNER_TOKEN_FARGATE_AGENT=$RUNNER_TOKEN_FARGATE_AGENT

RUN           prefect backend cloud

CMD           [ "prefect", "agent", "ecs", "start", "--token", "$RUNNER_TOKEN_FARGATE_AGENT", \
                    "--task-role-arn=arn:aws:iam::****:role/ECSTaskS3Role", \
                    "--log-level", "INFO", "--label", "fargate-dev", "--label", "s3-flow-storage", \
                    "--name", "fargate-demo" ]
n
We have deprecated the 
prefect auth create-token
command. You will need to get a new token. You can do that here
b
ah thanks @Natalie Smith - i will try that. I am not sure that is the root issue though, because I did have the whole setup running on Ec2 with the same token previously
n
Ok. From that error message it does like a token and permissions issue so let's see if creating a new one resolves the issue for you (and it's good to get a new token through our updated auth process.) If it still doesn't work please come back and let us know.
b
Yeah, I have changed it and still get the same error.
FYI when running it locally there seems to be more in the stack trace:
Copy code
2021-06-10 22:08:38,568] ERROR - fargate-demo | Failed to verify authentication.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/agent/agent.py", line 890, in _setup_api_connection
    self._verify_token(self.client.get_auth_token())
  File "/usr/local/lib/python3.8/site-packages/prefect/agent/agent.py", line 824, in _verify_token
    result = self.client.graphql(query="query { auth_info { api_token_scope } }")
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 311, in graphql
    raise AuthorizationError(result["errors"])
prefect.utilities.exceptions.AuthorizationError: [{'path': ['auth_info'], 'message': 'AuthenticationError: Forbidden', 'extensions': {'code': 'UNAUTHENTICATED'}}]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/agent.py", line 420, in start
    start_agent(ECSAgent, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/agent.py", line 103, in start_agent
    agent.start()
  File "/usr/local/lib/python3.8/site-packages/prefect/agent/agent.py", line 182, in start
    self._setup_api_connection()
  File "/usr/local/lib/python3.8/site-packages/prefect/agent/agent.py", line 894, in _setup_api_connection
    raise RuntimeError(
RuntimeError: Error while contacting API at <https://api.prefect.io>
ahh I got it
It was a docker issue for anyone out there, environment variables dont expand in the CMD unless you use a shell. Changed to this:
Copy code
CMD           [ "sh", "-c", "prefect agent ecs start --token $PREFECT__CLOUD__AGENT__AUTH_TOKEN \
                    --task-role-arn=arn:aws:iam::****:role/ECSTaskS3Role \
                    --log-level INFO --label fargate-dev --label s3-flow-storage \
                    --name fargate-demo" ]
n
Glad you got it working! Thanks for sharing
🙌 1
Open in Slack
PreviousNext
Powered by