https://prefect.io logo
l

Levi Leal

12/14/2020, 8:28 PM
Copy code
$ kubectl -n agent logs prefect-job-2d655498-sqqlc
[2020-12-14 20:20:37+0000] INFO - prefect.S3 | Downloading slow-flow/2020-12-07t12-16-09-863598-00-00 from lime-prefect
Unable to locate credentials
Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 34, in flow_run
    return _execute_flow_run()
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 99, in _execute_flow_run
    raise exc
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 84, in _execute_flow_run
    flow = storage.get_flow(storage.flows[flow_data.name])
  File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/s3.py", line 105, in get_flow
    self._boto3_client.download_fileobj(
  File "/usr/local/lib/python3.8/site-packages/boto3/s3/inject.py", line 678, in download_fileobj
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/usr/local/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/s3transfer/download.py", line 340, in _submit
    response = client.head_object(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 662, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 115, in create_request
    self._event_emitter.emit(event_name, request=request,
  File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/local/lib/python3.8/site-packages/botocore/signers.py", line 162, in sign
    auth.add_auth(request)
  File "/usr/local/lib/python3.8/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
Anyone who managed to make EKS Fargate work with prefect agents?
I've already tried to add the policy AmazonS3FullAccess on the Cluster Service Role but it didn't work. My desired solution would be something that avoids storing credentials. But it could be a last resort.
k

Kyle Moon-Wright

12/14/2020, 9:14 PM
Hey @Levi Leal, From the traceback, it looks like your FargateAgent is having trouble retrieving your flow code from S3 without the access credentials. If these aren’t provided, the Agent will look to environment variables to authenticate with boto3, which can alternatively be provided by utilizing your secrets manager of choice. Is this an option for your setup? How did you provide your flow code from S3 previously? As a sidenote, the FargateAgent is now deprecated for the ECSAgent for continued functionality (this was very recent) but is still viable.
l

Levi Leal

12/14/2020, 9:17 PM
I'm using the Kubernetes Agent. I have EKS (Amazon Kubernetes) with a Fargate profile. The agent is deployed normaly, but the job that tries to download the flow doesn't have access. I've tried to add S3 access to all roles on the cluster, but still doesn't work.
k

Kyle Moon-Wright

12/14/2020, 9:46 PM
Hmm, that’s a tough one to debug. This blurb was the only bit I could find, but I think you’ve already done this configuration:
Copy code
To enable S3 access by your kubernetes cluster on EKS, add the necessary permissions (AmazonS3FullAccess or AmazonS3ReadOnlyAccess) directly to the NodeInstanceRole used by aws-auth-cm.yaml after launching worker nodes and before applying aws-auth-cm.yaml with kubectl.
l

Levi Leal

12/15/2020, 9:40 AM
This is my current EKS compute config. I already did this one but it only works when I don't have the
fp-agent
profile as the jobs will be deployed to the Node Group.
When I create the profile, my agents and jobs are created by fargate. I created a IAM Role with S3 access policy and attached it to my Fargate Profile. I know the agents can connect to the pods on the nodegroup because I'm using a "cluster local DNS" on the agent and it is able to query new scheduled flows. But when the agent creates a new job on fargate, this job can't access S3.
4 Views