eddy davies
11/01/2022, 3:50 PMextraEnvVarsSecret: "aws-iam-key"
in the helm values.yaml and a k8s secret in the
apiVersion: v1
kind: Secret
metadata:
name: aws-iam-key
namespace: default
type: Opaque
stringData:
AWS_ACCESS_KEY_ID: "redacted"
AWS_SECRET_ACCESS_KEY: "redacted"
Mason Menges
11/01/2022, 4:06 PMJamie Zieziula
11/01/2022, 4:11 PMeddy davies
11/01/2022, 4:14 PMhelm install --values k8s-agent.yaml prefect-agent prefect/prefect-agent
where the --values
yaml is
agent:
image:
repository: prefecthq/prefect
prefectTag: 2-python3.10
pullPolicy: IfNotPresent
config:
workQueues:
- test
cloudApiConfig:
accountId: "redact"
workspaceId: "redact"
prefect api key added as secret as well
kubectl apply -f prefect-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: prefect-api-key
namespace: default
type: Opaque
stringData:
key: "redact"
then I added kube_config block with
from prefect.blocks.kubernetes import KubernetesClusterConfig
my_new_k8s_config = KubernetesClusterConfig.from_file(path="~/.kube/config")
my_new_k8s_config.save("my-new-k8s-config-block-name")
and select that from kubernetes-job/k8s-demo
and deploy to that
prefect deployment build log_flow.py:log_flow -n log-flow-s3 -sb s3/hubspot -q test -o log-flow-s3-k8s-deployment.yaml -t test -ib kubernetes-job/k8s-demo
Mason Menges
11/01/2022, 8:23 PMeddy davies
11/02/2022, 3:39 PMkubectl
with IAM Roles for Service Accounts but having some difficulties.
So installing s3fs is shown for ECS-Task section but not on KubernetesJob section, so I missed that, might be worth adding to docs. (I guess you could use remote storage without s3fs but that is an unlikely use case)Nate
11/02/2022, 8:48 PMeddy davies
11/03/2022, 10:56 AMagent:
image:
repository: my_repo/prefect_s3
Nate
11/03/2022, 3:43 PMSome form of remote storage is required thoughyep! that's correct - maybe we can make that more clear You shouldn't actually need s3fs in the pod running the agent specifically, because that pod is not actually pulling the flow code when the agent creates the pod for your flow run, the image that this new pod will use will need s3fs to pull flow code from s3 before running, so you can include that in the
image
that you give to your infrastructure block for your deployment (or pass {"EXTRA_PIP_PACKAGES": "s3fs"}
to the env of the infra block)
since the agent is just responsible for the submission of the flow runs to other pods on your cluster, you should be fine keeping the default image in the helm charteddy davies
11/04/2022, 2:38 PM14:02:51.647 | INFO | prefect.agent - Completed submission of flow run '2c74a2d0-b94f-4f4e-a3b9-1be3a0e936f3'
14:17:45.848 | INFO | prefect.agent - Submitting flow run 'da7ba311-720f-4194-bc1b-fa7a94c6c6f8'
/usr/local/lib/python3.10/site-packages/prefect/agent.py:215: UserWarning: Block document has schema checksum sha256:686f931093d8fa3a80dee6eb66516be7b022bf29c44a38766da8571f25fede8b which does not match the schema checksum for class 'KubernetesJob'. This indicates the schema has changed and this block may not load.
infrastructure_block = Block._from_block_document(infra_document)
14:17:46.302 | ERROR | root - [Errno 2] No such file or directory: 'aws-iam-authenticator'
14:17:46.321 | ERROR | prefect.agent - Failed to submit flow run 'da7ba311-720f-4194-bc1b-fa7a94c6c6f8' to infrastructure.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 259, in _submit_run_and_capture_errors
result = await infrastructure.run(task_status=task_status)
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 276, in run
job_name = await run_sync_in_worker_thread(self._create_job, manifest)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 68, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(call, cancellable=True)
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 505, in _create_job
job = batch_client.create_namespaced_job(self.namespace, job_manifest)
File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api/batch_v1_api.py", line 210, in create_namespaced_job
return self.create_namespaced_job_with_http_info(namespace, body, **kwargs) # noqa: E501
File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api/batch_v1_api.py", line 309, in create_namespaced_job_with_http_info
return self.api_client.call_api(
File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 391, in request
return <http://self.rest_client.POST|self.rest_client.POST>(url,
File "/usr/local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 276, in POST
return self.request("POST", url,
File "/usr/local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 235, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '3e57aac2-6c38-4f0b-80c2-6539f0d512e2', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '8ffede11-9091-4daa-8d22-0a80961f7f1f', 'X-Kubernetes-Pf-Prioritylevel-Uid': '7852fbbf-8622-427a-92e4-b6021eab94e3', 'Date': 'Fri, 04 Nov 2022 14:17:46 GMT', 'Content-Length': '290'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:anonymous\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
14:17:46.323 | INFO | prefect.agent - Completed submission of flow run 'da7ba311-720f-4194-bc1b-fa7a94c6c6f8'
rpc error: code = Unknown desc = Error: No such container: f8751f98ce4a07ae53d6cfe71633c904effc205b858ffb2c28be3795e1826500%
Emil Christensen
11/04/2022, 2:43 PM[Errno 2] No such file or directory: 'aws-iam-authenticator'
- not sure where this is coming from… maybe a secret? None of the flow code or flow code storage has anything with that name.
2. "status":"Failure","message":"jobs.batch is forbidden: User \"system:anonymous\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\""
- not sure why the use is anonymous
or creation of jobs is forbidden. Eddy is using our prefect-helm chart to deploy the agent.Nate
11/04/2022, 2:47 PMGeorge Coyne
11/04/2022, 2:50 PMAnna Geller
11/04/2022, 3:43 PMeddy davies
11/04/2022, 3:50 PMeksctl utils associate-iam-oidc-provider \
--region eu-west-2 \
--cluster eksdemo1 \
--approve
The cluster I am using now is called eksworkshop-eksctl
though so not sure that impacted it