https://prefect.io logo
Powered by
Title
s

sjammula

04/24/2023, 6:37 PM
Hi Team, I'm facing the following issue Flow code is not getting download from s3 location 
[notice] A new release of pip available: 22.3.1 -> 23.1.1
[notice] To update, run: pip install --upgrade pip
/usr/local/lib/python3.9/runpy.py:127: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
 warn(RuntimeWarning(msg))
17:10:58.262 | INFO  | Flow run 'berserk-camel' - Downloading flow code from storage at ''
17:11:00.383 | INFO  | prefect.engine - Engine execution of flow run 'dadf479a-39a5-4634-830b-66ad07276e64' aborted by orchestrator: This run has already terminated.
Any thoughts for this ? Here is my storage block code 
k8s_job = infrastructure.KubernetesJob(
    namespace="prefect",
    image="prefecthq/prefect:2-python3.9",
    image_pull_policy=infrastructure.KubernetesImagePullPolicy.IF_NOT_PRESENT,
    service_account_name="prefect",
    env={"EXTRA_PIP_PACKAGES": "s3fs"},
    overwrite=True,
)
Am I missing anything here?
✅ 1
s

Serina

04/24/2023, 7:08 PM
Hi @sjammula, a user faced a similar issue here that may perhaps be of help to you: https://prefect-community.slack.com/archives/CL09KU1K7/p1682099454008779
s

sjammula

04/25/2023, 6:50 PM
No @Serina G It didnot help me , I tried editing the "path" field to bypass that error where I was able to get through ,but now receiving No such file or directory though the file is present in the s3 bucket in the path given Can you assist on this pleas.
s

Serina

04/25/2023, 6:54 PM
Is the file called “hellow_world.py” or “hello_world.py”?
s

sjammula

04/25/2023, 7:08 PM
@Serina G hellow_world.py is the filename
s

Serina

04/25/2023, 7:19 PM
Are you able to replicate the issue locally? What does your yaml file look like?
s

sjammula

04/25/2023, 9:16 PM
@Serina G Attached my deployment.yaml file and also I receive this error sometime intermittently not sure when it happens though 'prefect' namespace is defined.Can you correct me where I'm going wrong. In the deployment.yaml I did provide my AWS accesskeys but removed in the attached file just for security reasons.Also edited "path" field manually as told you earlier. 
Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f1bcc3a2-e5f3-4c97-8f4a-414f11491e0d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '513745c7-aa7c-4245-8349-ec7b488ba2ba', 'X-Kubernetes-Pf-Prioritylevel-Uid': '17e7873d-25c0-45c1-955e-4c4692a6bb21', 'Date': 'Tue, 25 Apr 2023 21:11:31 GMT', 'Content-Length': '340'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect:prefect\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
deployment.yaml
Why do I see 403 error above in agent logs and different error "No such file or directory hello_world.py " in UI?
@Serina G any thoughts?
@prefect can I have thoughts pls on my above ask
n

Nate

04/26/2023, 6:10 PM
@sjammula hi, what namespace of your cluster is your agent running in? i.e. 
helm list | grep agent
if you used helm to deploy the agent
s

sjammula

04/26/2023, 6:15 PM
prefect
is the namespace in which agent is running @Nate
I did not install using helm
n

Nate

04/26/2023, 6:17 PM
this error is somewhat odd to me, seems like a cluster permissions issue with the service account 
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect:prefect\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
can you provide more of the agent logs during a failure?
👍 1
s

sjammula

04/26/2023, 6:18 PM
I gave * permission as well here apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: prefect name: prefect-agent-rbac rules: - apiGroups: [""] resources: ["pods", "pods/log", "pods/status"] verbs: ["get", "watch", "list"] - apiGroups: ["batch"] resources: ["jobs"] verbs: [ "*" ] ---
z

Zanie

04/26/2023, 6:23 PM
You also need 
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
  name: prefect-agent
rules:
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
:thank-you: 1
Take a look at the output of 
prefect kubernetes manifest agent
please
s

sjammula

04/26/2023, 6:25 PM
it is available @Nate
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prefect-agent rules: - apiGroups: [""] resources: ["namespaces"] verbs: ["get", "list"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prefect-agent-cluster-role-binding subjects: - kind: ServiceAccount name: default namespace: default roleRef: kind: ClusterRole name: prefect-agent apiGroup: rbac.authorization.k8s.io
looks like clusterrolebinding is pointing to default service account
@Nate do you think is that the issue?
n

Nate

04/26/2023, 6:57 PM
yes, now that you seem to have addressed Zanie's point, i believe that should be 
prefect
instead of 
default
(for both 
name
and 
namespace
)
s

sjammula

04/26/2023, 6:57 PM
can I edit that manually?
here is the log file
n

Nate

04/26/2023, 6:58 PM
ok yeah the error in those logs should be addressed by the change Zanie suggested
s

sjammula

04/26/2023, 6:59 PM
logs.txt
logs.txt
Does that resolve UI error which I 'm seeing in one of my first comment? 
17:10:58.262 | INFO  | Flow run 'berserk-camel' - Downloading flow code from storage at ''
n

Nate

04/26/2023, 6:59 PM
and yes you can edit the yaml config manually and then apply them, but I'll say that this sort of config is why the helm chart can be convenient
👀 1
s

sjammula

04/26/2023, 7:02 PM
in helm charts installation it is installing service account,role,rolebinding and then deployment.yaml but not clusterrole and clusterrolebinding right?
n

Nate

04/26/2023, 7:04 PM
Downloading flow code from storage at ''
this is somewhat misleading, this log sometimes appears even when the flow code is correctly downloaded from storage - we have an issue here for that
👀 1
and yes, I believe that this
it is installing service account,role,rolebinding and then deployment.yaml but not clusterrole and clusterrolebinding right?
is accurate
z

Zanie

04/26/2023, 7:05 PM
The helm chart infers the information we need from the system namespace at install time so it's not needed at runtime
👍 1
s

sjammula

04/26/2023, 7:23 PM
@Nate @Zanie I tried editing the 
path
manually on the generated deployment.yaml file and then applied it ,then noticed one more error with No such file or directory
I checked the bucket location where "hellow_world.py"exists in the given s3 path
n

Nate

04/26/2023, 7:39 PM
do you see your file here 
test-bucket/hellow_world.py
? if so, i don't believe you'd want to have your 
path
like that, or else I think it'd look at 
test-bucket/test-bucket/hellow_world.py
- I'd leave the 
path
empty in that case
s

sjammula

04/26/2023, 7:44 PM
I see the file in the location 
test-bucket/hellow_world.py
.Ok will not edit this path ok let me try by editing path back to ' ' and clusterrolebinding manually to 
prefect
namespace
@Nate I can edit the file generated from 
prefect kubernetes manifest agent
and then apply back right?
n

Nate

04/26/2023, 7:48 PM
yes, afaik about your scenario, that should work
s

sjammula

05/01/2023, 7:47 PM
Thanks @Nate and @Zanie.It worked after adding clusterrole and clusterrolebinding scoping to prefect namespace.
Can we perform the same steps in my prod environment?
2 Views
#prefect-communityJoin Slack
Powered by