What the distribution adds on top of Kubernetes — the operator ownership chain, projects, SCC admission and the control plane — and how each one changes where you look when production breaks.
Nothing reaches etcd without passing OAuth and admission first — which is why most OpenShift-specific failures are rejections, not crashes.
The layers below the API are all reconciled by operators, and the operators are reconciled by the CVO. That chain is the debugging path: ClusterVersion → ClusterOperator → operator Deployment → the resource it manages → Pods.
The diagram above is the high level: what the pieces are. These two are the ones you want when something is wrong — what is inside one of those boxes, and the path a request really takes through them.
The four things that make OpenShift behave differently from the Kubernetes you already know.
OpenShift is a conformant Kubernetes distribution. Every kubectl command works,
every Kubernetes object behaves the way the upstream docs say. What Red Hat adds is not a
different API — it is a managed lifecycle for the whole platform.
That distinction is the single most useful thing to internalise, because it tells you where
to look when something breaks. On vanilla Kubernetes, the cluster is whatever you assembled:
if the ingress controller is broken, you go and look at the ingress controller. On OpenShift,
almost every platform component is owned by an operator, and those operators are
owned by the Cluster Version Operator (CVO). A broken router is not just a broken
deployment — it is a ClusterOperator reporting Degraded=True, and the
CVO will keep reconciling it back to its declared state.
oc get clusteroperators
is the first command of any OpenShift incident — it tells you which of ~30 platform components
believes it is unhealthy, before you go looking at pods.The CVO is the root of the ownership tree. It reads the release payload — a container image holding every manifest for that exact version — and reconciles the cluster towards it. Each platform component gets a second-level operator; the CVO owns those operators, and they own their own workloads.
The chain is worth saying out loud because it is the debugging path:
ClusterVersion → CVO → ClusterOperator →
second-level operator Deployment → the workload it manages → Pods.
An upgrade "hangs". oc get clusterversion shows PROGRESSING=True
for an hour with no version change. The CVO applies manifests in order and stops at
the first one that will not become ready — so the answer is never "the upgrade is slow", it is
always "component N is not reconciling and everything behind it is queued".
A Project is a namespace with an OpenShift wrapper around it. The underlying
object really is a namespace — oc get namespace shows it — but creating one through
the Project API does more than create the namespace:
ResourceQuota,
a LimitRange, a default NetworkPolicy and RoleBindings automatically.
This is the hook to use when you want every new team namespace to arrive pre-governed.self-provisioner cluster role, which decides whether ordinary
users may create projects at all.Projects also gate what a user can see. A user with no access to a namespace gets a
Forbidden on the namespace API but an empty list from the Project API — which is
why the console shows a user only their own projects without leaking the names of others.
Creating the namespace directly with kubectl create namespace skips the template
entirely. The namespace exists, workloads run, and three months later someone notices this one
namespace has no quota and no default NetworkPolicy. If your platform relies on the project
template for guardrails, namespace creation must be restricted to the Project API.
An operator is a controller plus a CRD. The CRD gives you an object that describes intent
(I want a 3-node PostgreSQL with PITR
); the controller watches that object and does
whatever a human operator would do to make reality match — provision, configure, back up,
fail over, upgrade.
The part that matters operationally is the reconcile loop. It is level-driven, not edge-driven: the controller does not react to your change, it repeatedly compares desired state to actual state and corrects the difference. So any manual change to a managed resource has a half-life measured in seconds.
| System | Installs | Managed by |
|---|---|---|
| CVO | The ~30 core platform operators | Red Hat, tied to the cluster version |
| OLM (Operator Lifecycle Manager) | Optional and third-party operators from catalogs | You, via Subscription objects |
OLM adds its own vocabulary: a CatalogSource is a catalog of operators, a
Subscription says "install this one and keep it updated on this channel", an
InstallPlan is the pending upgrade, and a ClusterServiceVersion (CSV)
is the installed operator version. When an operator will not upgrade, the InstallPlan is
almost always where the answer is — most often waiting on a manual approval nobody gave.
Manual approval stops an operator upgrading itself into an incident. It also means an operator can sit months behind, quietly, with nothing alerting on it. If you choose Manual, alert on pending InstallPlans — otherwise you have chosen 'never upgrade' without deciding to.
Node configuration, admission policy, the image supply chain, and the one component that has no graceful degradation.
On OpenShift the node OS (RHCOS) is not configured by you logging in. It is configured by
MachineConfig objects, rendered by the Machine Config Operator into an Ignition
config, and applied by a per-node daemon that cordons, drains, writes, and reboots
the node.
MachineConfigs are grouped by MachineConfigPool — master and
worker by default. The MCO renders all MachineConfigs matching a pool into one
merged config and rolls it out node by node, respecting maxUnavailable.
A malformed MachineConfig does not fail at admission. It renders, rolls out to the first
node, and that node fails to come back. The pool then reports Degraded and
stops — which is the MCO protecting you. The cluster is now in a half-applied
state and the fix is to delete the offending MachineConfig and let the pool re-render.
SCC is OpenShift's pod-level admission policy, and it predates Kubernetes' own Pod Security Admission. It controls what a pod may ask for: running as root, host networking, host paths, privileged mode, which capabilities, which SELinux context, which UID range.
By default every authenticated user gets restricted-v2, which refuses root,
drops nearly all capabilities, and assigns a random high UID from the namespace's
range. That last part is what breaks third-party charts: an image with
USER 1000 and files owned by 1000 runs as UID 1000734512 instead and cannot write
to its own data directory.
The give-away is a pod that will not schedule with a message naming SCC, or a pod that starts and immediately fails on permissions. The annotation on a running pod tells you which SCC actually admitted it.
Granting anyuid to the service account makes the error go away and hands the workload the right to run as root. Fix the image instead: make the data directory group-writable and owned by GID 0, which is what OpenShift-compatible images do — the random UID is always in group 0. When you genuinely need elevated access, create a custom SCC granting only the specific capability, and bind it to one service account.
An ImageStream is a pointer to images, not a store of them. Each tag resolves
to an immutable digest, and the stream records the history of what that tag pointed at. That
gives you two things vanilla Kubernetes does not have out of the box: a rollback
target, and a trigger — a Deployment can be told to redeploy when a
stream tag moves.
BuildConfig is the in-cluster build. Source-to-Image (S2I) takes application
source plus a builder image and produces a runnable image without a Dockerfile; Docker strategy
builds a Dockerfile; Custom runs your own builder image.
:latest in an external registry resolves
once, at import. It does not follow upstream unless scheduled: true is set on the
tag — so "we pushed a new latest and nothing happened" is expected behaviour, not a bug.Every object in the cluster is etcd state. etcd is a Raft cluster of 3 (or 5) members and needs a strict majority to accept writes: 3 members tolerate 1 failure, 5 tolerate 2. Lose quorum and the API server goes read-only — the cluster does not "run degraded", it stops accepting change.
etcd is also brutally sensitive to disk latency, because every write is fsynced before it is acknowledged. The practical threshold is a 99th-percentile fsync under ~10 ms. Above that you get leader elections, and leader elections during an upgrade produce the "everything is slow and nothing is broken" incident.
oc debug node/<master> then /usr/local/bin/cluster-backup.sh produces a snapshot and the static pod manifests. Restoring one is a documented but genuinely disruptive procedure that takes the cluster down and rolls every node. Do it once, on a cluster you can afford to break, before you need it.
OpenShift gives you one entry point for almost any platform failure, and it is not
oc get pods. Work down the ownership chain — cluster operator, then the operator's
own workload, then the resource it manages.
| Symptom | Where it actually is | First command |
|---|---|---|
| Console unreachable, API fine | Ingress operator or the router pods | oc get co ingress -o yaml |
oc login fails for everyone | Authentication operator / OAuth pods / the IdP itself | oc get co authentication; oc logs -n openshift-authentication -l app=oauth-openshift |
| Pods Pending, nodes look fine | Scheduler constraints, or a quota on the project | oc describe pod — the Events tail names the predicate that failed |
| Pods rejected at creation | SCC admission | oc adm policy scc-subject-review -z <sa> -f <file> |
| Node NotReady, no obvious cause | MCO mid-rollout, or kubelet/CRI-O on the node | oc get mcp; then oc debug node/<n> -- chroot /host journalctl -u kubelet -n 200 |
| Upgrade stalled at N% | The cluster operator named in the Progressing message | oc get clusterversion -o yaml |
| Everything slow, nothing down | etcd fsync latency or a leader election storm | oc logs -n openshift-etcd etcd-<master> -c etcd | grep -i 'elected\|slow' |
| Operator stuck on an old version | An unapproved InstallPlan | oc get installplan -A |
oc adm must-gather is the supported way to capture cluster state. Run it with no
arguments and you get the full platform dump; point it at a component image and you get that
component's deep state instead. The full collection can run to several GB and take 15+ minutes,
so scope it when you already know the area.
On OpenShift the useful message is nearly always in oc describe output or oc get events --sort-by=.lastTimestamp, not in a container log. Admission rejections, scheduling failures, image pull errors, quota denials and SCC refusals are all Events — none of them ever reach a pod log, because the pod never started.
| Command | What it answers |
|---|---|
oc get clusteroperators | Which of the ~30 platform components is unhealthy — start here |
oc get clusterversion | Current version, and what an in-flight upgrade is waiting on |
oc get mcp | Whether node config is mid-rollout or wedged |
oc get nodes -o wide | Node state, roles, kernel and runtime versions |
oc get events -A --sort-by=.lastTimestamp | tail -40 | What the cluster just complained about |
oc describe pod <p> | Scheduling, admission and image-pull failures |
oc get pod <p> -o yaml | grep scc | Which SCC admitted it |
oc adm policy scc-subject-review -z <sa> -f f.yaml | Which SCC would admit a workload |
oc get subscription,installplan,csv -A | OLM operator state end to end |
oc adm top nodes | Actual node CPU/memory pressure |
oc debug node/<n> -- chroot /host journalctl -u kubelet | Node-level logs without SSH |
oc adm must-gather --dest-dir=./mg | The supported full cluster dump |
oc get project | Projects you can see (never leaks ones you cannot) |
oc status -n <ns> | A readable summary of what is running in a project |