In the work pool -> worker page for the work pool of our interests, we can clearly see that the worker I started in our own kubernetes environment via helm-chart has successfully connected to the Prefect UI, with a green light & showing that the last time it was seen is within minutes. The worker's k8s log also says:

Worker 'KubernetesWorker <worker id>' started!

However, if we start a job from prefect ui for this work pool, unless the job is started in the "default" queue, the job will not run, only shows that it's "late". Also, even for cases where the job is in the "default" queue, apparently, these job will run till the end, but I can't see anything in the worker's log saying that the worker had pickuped the job and track them till finished. The worker was started with the work queues array including all 3 queue type we specified, so it should be able to fetch jobs from any of the queue we selected. One thing strange we found out is that when we try to cancel the job that shows "late", it only ends up in "cancelling" state, not "cancelled". But this would end up in showing some logs in the worker, saying:

Found 1 flow runs awaiting cancellation.
Skipping cancellation because flow run '<the id we are trying to cancel>' is using enhanced cancellation. A dedicated runner will handle cancellation

I resolved this by setting https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-worker/values.yaml#L16C3-L16C13 I realize I didn't set this value at all, not even to empty string. My guess is that this means the value on my side will be null, and null resullted in this issues.