https://prefect.io logo
Join Slack
Powered by
Hi Team, I'm facing the following issue Flow code ...
# ask-community
s
Hi Team, I'm facing the following issue Flow code is not getting download from s3 location
Copy code
[notice] A new release of pip available: 22.3.1 -> 23.1.1
[notice] To update, run: pip install --upgrade pip
/usr/local/lib/python3.9/runpy.py:127: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
 warn(RuntimeWarning(msg))
17:10:58.262 | INFO  | Flow run 'berserk-camel' - Downloading flow code from storage at ''
17:11:00.383 | INFO  | prefect.engine - Engine execution of flow run 'dadf479a-39a5-4634-830b-66ad07276e64' aborted by orchestrator: This run has already terminated.
Any thoughts for this ? Here is my storage block code
Copy code
k8s_job = infrastructure.KubernetesJob(
    namespace="prefect",
    image="prefecthq/prefect:2-python3.9",
    image_pull_policy=infrastructure.KubernetesImagePullPolicy.IF_NOT_PRESENT,
    service_account_name="prefect",
    env={"EXTRA_PIP_PACKAGES": "s3fs"},
    overwrite=True,
)
Am I missing anything here?
✅ 1
s
Hi @sjammula, a user faced a similar issue here that may perhaps be of help to you: https://prefect-community.slack.com/archives/CL09KU1K7/p1682099454008779
s
No @Serina G It didnot help me , I tried editing the "path" field to bypass that error where I was able to get through ,but now receiving No such file or directory though the file is present in the s3 bucket in the path given Can you assist on this pleas.
s
Is the file called “hellow_world.py” or “hello_world.py”?
s
@Serina G hellow_world.py is the filename
s
Are you able to replicate the issue locally? What does your yaml file look like?
s
@Serina G Attached my deployment.yaml file and also I receive this error sometime intermittently not sure when it happens though 'prefect' namespace is defined.Can you correct me where I'm going wrong. In the deployment.yaml I did provide my AWS accesskeys but removed in the attached file just for security reasons.Also edited "path" field manually as told you earlier.
Copy code
Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f1bcc3a2-e5f3-4c97-8f4a-414f11491e0d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '513745c7-aa7c-4245-8349-ec7b488ba2ba', 'X-Kubernetes-Pf-Prioritylevel-Uid': '17e7873d-25c0-45c1-955e-4c4692a6bb21', 'Date': 'Tue, 25 Apr 2023 21:11:31 GMT', 'Content-Length': '340'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect:prefect\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
deployment.yaml
Why do I see 403 error above in agent logs and different error "No such file or directory hello_world.py " in UI?
@Serina G any thoughts?
@prefect can I have thoughts pls on my above ask
n
@sjammula hi, what namespace of your cluster is your agent running in? i.e. 
helm list | grep agent
if you used helm to deploy the agent
s
prefect
is the namespace in which agent is running @Nate
I did not install using helm
n
this error is somewhat odd to me, seems like a cluster permissions issue with the service account
Copy code
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect:prefect\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
can you provide more of the agent logs during a failure?
👍 1
s
I gave * permission as well here apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: prefect name: prefect-agent-rbac rules: - apiGroups: [""] resources: ["pods", "pods/log", "pods/status"] verbs: ["get", "watch", "list"] - apiGroups: ["batch"] resources: ["jobs"] verbs: [ "*" ] ---
z
You also need
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
  name: prefect-agent
rules:
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
🙏 1
Take a look at the output of 
prefect kubernetes manifest agent
please
s
it is available @Nate
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prefect-agent rules: - apiGroups: [""] resources: ["namespaces"] verbs: ["get", "list"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prefect-agent-cluster-role-binding subjects: - kind: ServiceAccount name: default namespace: default roleRef: kind: ClusterRole name: prefect-agent apiGroup: rbac.authorization.k8s.io
looks like clusterrolebinding is pointing to default service account
@Nate do you think is that the issue?
n
yes, now that you seem to have addressed Zanie's point, i believe that should be 
prefect
instead of 
default
(for both 
name
and 
namespace
)
s
can I edit that manually?
here is the log file
n
ok yeah the error in those logs should be addressed by the change Zanie suggested
s
logs.txt
logs.txt
Does that resolve UI error which I 'm seeing in one of my first comment?
Copy code
17:10:58.262 | INFO  | Flow run 'berserk-camel' - Downloading flow code from storage at ''
n
and yes you can edit the yaml config manually and then apply them, but I'll say that this sort of config is why the helm chart can be convenient
👀 1
s
in helm charts installation it is installing service account,role,rolebinding and then deployment.yaml but not clusterrole and clusterrolebinding right?
n
Copy code
Downloading flow code from storage at ''
this is somewhat misleading, this log sometimes appears even when the flow code is correctly downloaded from storage - we have an issue here for that
👀 1
and yes, I believe that this
it is installing service account,role,rolebinding and then deployment.yaml but not clusterrole and clusterrolebinding right?
is accurate
z
The helm chart infers the information we need from the system namespace at install time so it's not needed at runtime
👍 1
s
@Nate @Zanie I tried editing the 
path
manually on the generated deployment.yaml file and then applied it ,then noticed one more error with No such file or directory
I checked the bucket location where "hellow_world.py"exists in the given s3 path
n
do you see your file here 
test-bucket/hellow_world.py
? if so, i don't believe you'd want to have your 
path
like that, or else I think it'd look at 
test-bucket/test-bucket/hellow_world.py
- I'd leave the 
path
empty in that case
s
I see the file in the location 
test-bucket/hellow_world.py
.Ok will not edit this path ok let me try by editing path back to ' ' and clusterrolebinding manually to 
prefect
namespace
@Nate I can edit the file generated from 
prefect kubernetes manifest agent
and then apply back right?
n
yes, afaik about your scenario, that should work
👍 1
s
Thanks @Nate and @Zanie.It worked after adding clusterrole and clusterrolebinding scoping to prefect namespace.
Can we perform the same steps in my prod environment?
3 Views
Open in Slack
PreviousNext
Powered by