ConfigMaps that update in place except when they do not, Secrets that are encoded rather than encrypted, namespaces that isolate less than people assume, and RBAC's one rule — purely additive, no deny — that explains every access surprise.
Authentication, authorisation, admission, then quota — in that order, each refusing in its own words.
Only the bottom half is namespaced. ClusterRoles, nodes and PersistentVolumes sit outside any namespace, which is the source of most "but I gave them access" confusion.
The diagram above is the high level: what the pieces are. These two are the ones you want when something is wrong — what is inside one of those boxes, and the path a request really takes through them.
The four objects every cluster uses, and the property of each that is not what people assume.
A ConfigMap is a key/value object consumed in one of two ways, and the two behave
completely differently at runtime:
| Consumed as | On ConfigMap update |
|---|---|
env / envFrom | Never changes. Environment is set once at container start. |
| Volume mount | Updated in place, typically within a minute (kubelet sync period). |
Volume mount with subPath | Never changes. subPath breaks the symlink swap the update relies on. |
"We updated the ConfigMap and nothing happened" is almost always row 1 or row 3. Neither is a bug; both are documented; both are invisible unless you know to look.
If the app cannot reload config itself, the honest answer is to restart it — and the tidy way is to put a hash of the config in the pod template annotation, so changing the ConfigMap changes the Deployment and triggers a normal rolling update.
A Secret is a ConfigMap with a different name and base64-encoded values.
base64 is encoding, not encryption — anyone who can read the object can read the
value, and by default the bytes sit in etcd in the clear.
Three things actually protect a Secret, and all three are opt-in:
EncryptionConfiguration on the API
server encrypts Secrets before they reach etcd — ideally with a KMS provider rather than a local
key that sits next to the data it protects.get secrets in a namespace means every secret in it.
There is no per-object RBAC in core Kubernetes, so "read one secret" is really "read them all"
unless you split namespaces."Who can read the database password?" is an RBAC query, not a Secret query — and it has a precise answer.
Sealed Secrets, SOPS or an external store (Vault, a cloud secret manager via the Secrets Store CSI driver) all solve this. Base64 in a committed manifest solves nothing — it is a rendering choice, not a security boundary, and the repo history keeps it after you delete the file.
A namespace scopes names, and gives you something to attach RBAC,
ResourceQuota and LimitRange to. That is genuinely useful, and it is
also the whole list.
What a namespace does not isolate, and each has bitten someone:
So "we put the untrusted tenant in its own namespace" is not a security statement by itself. It becomes one when you add a default-deny NetworkPolicy, a quota, a restricted Pod Security Admission level and — for genuinely untrusted workloads — separate nodes or a separate cluster.
Four objects, two axes. Role and RoleBinding are namespaced;
ClusterRole and ClusterRoleBinding are not:
| Combination | Grants |
|---|---|
| Role + RoleBinding | Those verbs in that one namespace |
| ClusterRole + RoleBinding | The ClusterRole's verbs, but only in the binding's namespace — the useful one people forget |
| ClusterRole + ClusterRoleBinding | Every namespace, plus cluster-scoped resources |
| Role + ClusterRoleBinding | Invalid. Silently grants nothing. |
RBAC is purely additive and has no deny. Effective permission is the union of every binding that matches you. So you cannot subtract a permission with another rule — you have to find and remove the binding that granted it. And a user in three groups has all three groups' permissions, which is why "I thought we removed their access" is usually a second binding nobody looked for.
Several innocuous-looking verbs are effectively cluster-admin:
create pods (mount any secret, or a hostPath),
escalate/bind (grant yourself more),
impersonate (become anyone), and
get secrets where a privileged ServiceAccount token lives.
Access failures and config failures look identical from the outside — the pod does not work — and are diagnosed completely differently. The refusal wording tells you which gate said no.
| What you see | Which gate | Next command |
|---|---|---|
Unauthorized / 401 | Authentication — the identity did not resolve | kubectl auth whoami; check the kubeconfig context |
Forbidden: User "x" cannot <verb> | RBAC | kubectl auth can-i --list --as=x |
violates PodSecurity "restricted" | Pod Security Admission | kubectl get ns <n> -o jsonpath='{.metadata.labels}' |
exceeded quota | ResourceQuota | kubectl describe quota -n <n> |
admission webhook ... denied | A validating webhook | kubectl get validatingwebhookconfigurations |
| Pod runs, config is stale | env vars or a subPath mount | kubectl exec -- env; then the volume's subPath |
CreateContainerConfigError | A referenced ConfigMap or Secret is missing | kubectl describe pod names the key it could not find |
A validating or mutating webhook with failurePolicy: Fail whose backing service
is unavailable rejects every matching API write. If its own namespace is in scope, you
cannot deploy the fix — including the webhook itself. This is a genuine "cannot deploy anything"
outage and the escape is to delete the webhook configuration.
Reading Roles and bindings by hand gets the answer wrong, because effective permission is the union of every matching binding including ones granted through groups. kubectl auth can-i --list --as=<subject> asks the API server to do the resolution it will actually do.
| Command | What it answers |
|---|---|
kubectl auth can-i --list -n <ns> --as=<user> | Effective permissions, resolved by the API server |
kubectl auth can-i <verb> <res> --as=system:serviceaccount:<ns>:<sa> | The service-account form |
kubectl auth whoami | Which identity this kubeconfig presents |
kubectl get clusterrolebindings -o json | jq ... cluster-admin | Who holds the keys |
kubectl describe quota -n <ns> | Used versus hard, per resource |
kubectl get ns <n> -o jsonpath='{.metadata.labels}' | Which Pod Security level is enforced |
kubectl exec deploy/<x> -- env | What the container actually received |
kubectl get cm,secret -n <ns> | What exists to be referenced |
kubectl create secret generic x --from-literal=k=v --dry-run=client -o yaml | Generate without applying |
kubectl get validatingwebhookconfigurations | What can reject your writes |
kubectl rollout restart deploy/<x> | Pick up changed env-var config |