mTLS identity that makes policy a statement about services rather than subnets, the AuthorizationPolicy default that flips one workload to deny, retries that multiply through a call graph, and the trace headers your app still has to forward itself.
Once each workload has a cryptographic identity, authorization and routing stop being about IP addresses.
Telemetry sits at the bottom because it is a by-product: everything already passes through the proxy, so metrics come free — and traces do not, because they need the application to cooperate.
The diagram above is the high level: what the pieces are. These two are the ones you want when something is wrong — what is inside one of those boxes, and the path a request really takes through them.
The four things a mesh is actually run for, and where each one bites.
The mesh issues every workload a short-lived X.509 certificate whose identity is a SPIFFE ID derived from its ServiceAccount:
spiffe://cluster.local/ns/prod/sa/payments-sa
Both ends present one, both verify, and rotation is automatic and frequent — typically hourly. That is the part worth appreciating: certificate rotation, the thing that causes outages everywhere else, becomes invisible infrastructure.
And crucially, this identity is cryptographic, not network-based. Policy can now say "payments may call ledger" rather than "10.4.0.0/16 may reach port 8080" — which survives pods moving, IPs changing and namespaces being recreated.
PERMISSIVE accepts both mTLS and plaintext, which is what makes incremental
adoption possible. It is also indistinguishable from STRICT when everything happens to be
meshed — so clusters sit in PERMISSIVE for years believing they have mutual TLS, while any
unmeshed pod can still connect in plaintext.
With identity established, authorization becomes a statement about services rather than addresses. The evaluation order is worth committing to memory, because it is where surprises come from:
Point 4 is the one that catches people. A namespace with no AuthorizationPolicy is fully open. Adding one ALLOW policy to a single workload flips that workload to default-deny while everything beside it stays open — which is usually intended, and rarely realised.
An ALLOW policy with an empty spec: {} matches nothing and therefore denies
everything in its namespace. That is the idiom for a default-deny floor, and it reads as a typo
if you have not seen it before.
A debug pod carries its own ServiceAccount, so it has a different SPIFFE ID and will be refused for reasons that have nothing to do with the policy you are testing. Exec into an actual client pod, or run the probe with the same ServiceAccount.
VirtualService decides routing — match on header, path or weight;
DestinationRule defines the subsets and the upstream policy. A canary is a weight
change, and a header-matched route lets you send only your own traffic to the new version first,
which is a better first step than 1% of everyone.
Mesh retries are per-hop, and they multiply through a call graph. Three retries at each of three hops is up to 27 requests hitting the service at the bottom — so a service that is struggling receives an order of magnitude more load precisely because it started failing. This is the mechanism behind a large share of mesh-amplified outages.
Three rules keep it safe: retry only idempotent operations, keep
perTryTimeout × attempts under the overall timeout, and pair retries with
outlier detection so a consistently failing endpoint is ejected rather than
retried at.
Because every request crosses a proxy, the mesh reports request rate, error rate and latency distribution for every service-to-service hop, labelled with both identities, with no application instrumentation at all. That is the fastest observability win available in a Kubernetes estate, and it arrives the day you install the mesh.
Distributed tracing is not free. The mesh generates and forwards span
context, but the application must propagate the trace headers
(traceparent, or the x-b3-* family) from the incoming request to its
outgoing calls. An app that does not becomes a wall: every trace stops there and you get
disconnected fragments rather than a call graph. It is a handful of lines in most frameworks, and
it is the single most common reason mesh tracing "does not work".
Two shapes. Multi-primary puts a control plane in each cluster — no cross-cluster control dependency, more to keep consistent. Primary-remote has one control plane serving several clusters — simpler config, and a hard dependency on the primary. Either way the prerequisites are the same and are the actual work: a shared trust root so identities verify across clusters, pod-to-pod reachability or east-west gateways, and consistent namespace and ServiceAccount naming, because identity is derived from those names.
Operational mesh failures split cleanly in two: policy refused it, which is loud and precise, or routing sent it nowhere, which shows as a 503 with a two-character flag. Identify which before touching any YAML.
| What you see | Meaning | Where to look |
|---|---|---|
RBAC: access denied | AuthorizationPolicy refused it | Proxy log names the policy; check the caller's SPIFFE ID |
503 UF | Could not connect upstream | Endpoints, and mTLS mode mismatch |
503 UH | No healthy upstream — all ejected | Outlier detection is doing its job |
503 NR | No route matched | VirtualService hosts and ports |
503 UO | Circuit breaker open | connectionPool limits |
| Load spike on a failing service | Retries amplifying through the graph | attempts per hop, multiplied |
| Traces stop at one service | That app does not forward trace headers | Its outgoing request headers |
| Cross-cluster calls fail | Trust root or gateway | istioctl proxy-config endpoint — remote endpoints present? |
A STRICT namespace and an unmeshed caller produce a reset with no useful application-level error. It is easy to misread as a network problem. The tell: it works from inside the mesh and fails from outside, and the server-side proxy log shows the connection being closed before any request line.
It catches the majority of real-world mesh misconfiguration statically — hosts that do not resolve, conflicting policies, subsets with no matching DestinationRule, gateways selecting nothing. Running it in CI against your manifests catches these before they reach a cluster at all.
| Command | What it answers |
|---|---|
istioctl analyze -A | Static misconfiguration across the mesh |
istioctl x describe pod <p> | Every policy in force for one workload |
kubectl get peerauthentication -A | Which namespaces are actually STRICT |
kubectl get authorizationpolicy -A | Who may call whom |
kubectl logs <p> -c istio-proxy | grep rbac | Which policy refused a request |
kubectl logs <p> -c istio-proxy | grep ' 503 ' | The response flag — the actual diagnosis |
istioctl proxy-config endpoint <p> | Whether there is anything to route to |
istioctl proxy-config route <p> | Whether a route matches at all |
istioctl pc secret <p> | The workload cert and its validity |
linkerd viz stat deploy -n <ns> | Success rate, RPS, p99 per workload |
linkerd viz tap deploy/<x> | Live per-request stream |
istioctl proxy-status | Stale config, before you debug the wrong thing |