sjammula
04/24/2023, 6:37 PM[notice] A new release of pip available: 22.3.1 -> 23.1.1
[notice] To update, run: pip install --upgrade pip
/usr/local/lib/python3.9/runpy.py:127: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
17:10:58.262 | INFO | Flow run 'berserk-camel' - Downloading flow code from storage at ''
17:11:00.383 | INFO | prefect.engine - Engine execution of flow run 'dadf479a-39a5-4634-830b-66ad07276e64' aborted by orchestrator: This run has already terminated.
Any thoughts for this ?
Here is my storage block code
k8s_job = infrastructure.KubernetesJob(
namespace="prefect",
image="prefecthq/prefect:2-python3.9",
image_pull_policy=infrastructure.KubernetesImagePullPolicy.IF_NOT_PRESENT,
service_account_name="prefect",
env={"EXTRA_PIP_PACKAGES": "s3fs"},
overwrite=True,
)
Am I missing anything here?Serina
04/24/2023, 7:08 PMsjammula
04/25/2023, 6:50 PMSerina
04/25/2023, 6:54 PMsjammula
04/25/2023, 7:08 PMSerina
04/25/2023, 7:19 PMsjammula
04/25/2023, 9:16 PMSubmission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f1bcc3a2-e5f3-4c97-8f4a-414f11491e0d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '513745c7-aa7c-4245-8349-ec7b488ba2ba', 'X-Kubernetes-Pf-Prioritylevel-Uid': '17e7873d-25c0-45c1-955e-4c4692a6bb21', 'Date': 'Tue, 25 Apr 2023 21:11:31 GMT', 'Content-Length': '340'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect:prefect\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
Nate
04/26/2023, 6:10 PMhelm list | grep agent
if you used helm to deploy the agentsjammula
04/26/2023, 6:15 PMprefect
is the namespace in which agent is running @NateNate
04/26/2023, 6:17 PMHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect:prefect\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
can you provide more of the agent logs during a failure?sjammula
04/26/2023, 6:18 PMZanie
04/26/2023, 6:23 PMapiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
name: prefect-agent
rules:
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
prefect kubernetes manifest agent
pleasesjammula
04/26/2023, 6:25 PMNate
04/26/2023, 6:57 PMprefect
instead of default
(for both name
and namespace
)sjammula
04/26/2023, 6:57 PMNate
04/26/2023, 6:58 PMsjammula
04/26/2023, 6:59 PM17:10:58.262 | INFO | Flow run 'berserk-camel' - Downloading flow code from storage at ''
Nate
04/26/2023, 6:59 PMsjammula
04/26/2023, 7:02 PMNate
04/26/2023, 7:04 PMDownloading flow code from storage at ''
this is somewhat misleading, this log sometimes appears even when the flow code is correctly downloaded from storage - we have an issue here for thatit is installing service account,role,rolebinding and then deployment.yaml but not clusterrole and clusterrolebinding right?is accurate
Zanie
04/26/2023, 7:05 PMsjammula
04/26/2023, 7:23 PMpath
manually on the generated deployment.yaml file and then applied it ,then noticed one more error with No such file or directoryNate
04/26/2023, 7:39 PMtest-bucket/hellow_world.py
? if so, i don't believe you'd want to have your path
like that, or else I think it'd look at test-bucket/test-bucket/hellow_world.py
- I'd leave the path
empty in that casesjammula
04/26/2023, 7:44 PMtest-bucket/hellow_world.py
.Ok will not edit this path
ok let me try by editing path back to ' ' and clusterrolebinding manually to prefect
namespaceprefect kubernetes manifest agent
and then apply back right?Nate
04/26/2023, 7:48 PMsjammula
05/01/2023, 7:47 PM