Health Endpoint Monitoring

The problem

A platform that runs several instances of a service has to keep deciding three things about each one: has it finished starting, is it still working, and should it receive traffic right now. Left to itself it can only guess from the outside: the process exists, the port accepts connections. That guess is often wrong. A process can be up and useless: still warming its cache, out of database connections, deadlocked, or already shutting down.

Each wrong guess costs something different. Traffic sent to an instance that isn't ready becomes errors for users. Restarting an instance that was only busy throws away its warm state and pushes its load onto the others. And when every instance reports that it is fine, nobody may notice that users can't reach the service at all, because what broke is the DNS record, the certificate or the load balancer in front.

How it works

The service reports its own state through small, cheap endpoints, one per question, and a different consumer acts on each answer.

The split matters because the remedies differ. Taking an instance out of rotation is cheap and reversible, so readiness can react quickly. A restart is drastic and repairs nothing outside the process, so liveness should be slow to act and blind to everything but the process. In the diagram the endpoints are /startupz, /livez and /readyz. The last two follow a common convention; the first name is this diagram's own.

Who asks depends on the platform. In Kubernetes the kubelet runs all three probes on the node. It holds the liveness and readiness probes back until the startup probe has passed, and a failed readiness probe takes the Pod out of the endpoints of the Services that select it. A load balancer that probes on its own schedule knows nothing about the startup check: it simply keeps getting not ready until the instance has started.

Shallow and deep checks, and the dependency trap

A shallow check answers from what the process already knows: it is running, its workers are turning over, its pool holds connections. A deep check goes out and exercises a dependency, for example by running a query or calling a downstream API.

Deep checks find more, and they fail in a dangerous way. When the dependency is shared, every instance's check fails at the same moment:

• In a liveness check, the orchestrator restarts every instance at once. No restart repairs the database, and the service comes back cold into the same outage.
• In a readiness check, the load balancer loses every target and has nowhere to send traffic. Requests that never needed the database (cached reads, static content, other endpoints) fail too, so a partial problem becomes a total one.

So decide check by check:

• Readiness may include the instance's own resources: its connection pool is established, its cache is warm, the configuration and secrets it needs are loaded, its request queue or thread pool is not saturated, and it has not begun shutting down. If one of these is wrong, another instance really can do better, and that is exactly what taking this one out of rotation assumes.
• Leave out dependencies that every instance shares, such as the main database or a message broker. Removing instances helps only when the remaining ones are better off. Fail the individual requests that need the dependency, keep serving what still works, and let the dependency's own alarms report it.
• Leave out downstream services that have their own health checks and alarms. Guard those calls with timeouts and a Circuit Breaker, and degrade the feature, not the instance.
• Liveness includes none of these.

This is a judgement call, and good sources draw the line in slightly different places. The Kubernetes documentation allows a readiness probe to check the back-end services an application strictly depends on, so that traffic doesn't go to Pods that could only answer with errors. Spring Boot's documentation says the choice has to be made carefully and names the dependency shared by all instances as the hard case. If you do include a shared dependency, make sure something limits the damage: a balancer that fails open when every target is unhealthy, or a cap on the share of targets it may take out.

Steps 2 and 4 of the diagram are the two sides of that line. In step 2 instance A has lost its own connections while the database and the other instances are fine, so taking A out of rotation helps. In step 4 the database is slow for everyone, and removing instances helps nobody. The line is not always that clean: when a shared database is down completely, every instance's pool empties, and even a shallow pool check fails everywhere. That case is what failing open is for.

The answer

• Status code first: 200 for healthy and 503 Service Unavailable for not healthy. Machines should read nothing else. Kubernetes counts any status from 200 to 399 as a pass.
• A small body for humans: an overall status and, at most, the name and state of each check. Mark the response as not cacheable, so that no proxy answers on the instance's behalf. There is no standard body format; an Internet-Draft that proposed one expired in 2022 without becoming an RFC.
• No secrets. Connection strings, internal host names, version numbers and stack traces help an attacker more than an operator. Serve details only to authenticated callers or on an internal port, and keep the endpoint that the balancer probes minimal. A path that is hard to guess is not access control.
• gRPC services use the standard health service instead of a path: grpc.health.v1.Health. Its Check call returns SERVING or NOT_SERVING for a named service (the empty name stands for the whole server), and its Watch call streams the changes.

Intervals, timeouts and thresholds

• Detection takes about interval × failure threshold, plus the timeout of the last probe. With the Kubernetes defaults (a probe every 10 seconds, a 1-second timeout, 3 failures in a row) that is roughly half a minute.
• One failed probe should rarely decide anything. Requiring several failures in a row, and several passes before an instance returns, keeps a struggling instance from flapping in and out of rotation.
• Give liveness more patience than readiness, so that an instance is out of rotation for a while before anything kills it. The Kubernetes documentation describes using one cheap endpoint for both, with a higher failure threshold on the liveness probe.
• A slow boot is not a hang. A startup check with its own generous budget (failure threshold × period) covers initialisation, so the liveness check can stay tight afterwards.
• The numbers in the diagram are illustrative: one answer moves an instance in or out of rotation, and three missed probes restart it.

Keep the checks cheap

• Answer from state the process already holds and keeps fresh in the background, not by doing work for every probe. Probes arrive from every balancer node, the orchestrator and every monitor, all the time. A check that runs a query per probe sends a steady stream of queries at a database that may already be struggling.
• If a check has to call something, give the call a short timeout and consider caching the result for a few seconds. The cache trades load for staleness, so size it against how fast you need to detect a failure.
• Under overload the health endpoint must still answer. If busy instances time out on their probes, the balancer ejects them, their load lands on the rest, and those fail next: the health checking itself spreads the failure. Reserve capacity for the health endpoint, and shed real requests before the probes start to time out.
• Where you can, serve the check on the same port and through the same server stack as real traffic. A check on a separate management port can pass while the main listener refuses connections.

Shutting down

An instance that is about to stop should fail readiness first, keep serving the requests it already has and the few that still arrive, wait until the balancer has noticed, and only then exit. Liveness keeps passing the whole time, because the process is doing exactly what it should. Google's SRE book calls this the lame duck state. The wait has to cover the balancer's detection time or its deregistration delay: see Load Balancing for draining and Rolling Update for how a rollout depends on it. On Kubernetes, deleting a Pod already marks its endpoint as not ready, so the readiness endpoint does not have to flip for that case, but the process still has to keep serving for a moment after SIGTERM.

From the outside in

Every instance can be healthy while users see errors: an expired certificate, a wrong DNS record, a misconfigured balancer, or a network path that internal probes never take. An external monitor calls the public endpoint the way a user would, from several locations, so that one bad network path is not mistaken for an outage. It checks the content and the response time, not just the status code. Its results feed alerts, regional failover in a Multi-Region Active-Active setup, and the availability figures behind SLOs & Error Budgets. Probes are only a sample, though. Where there is enough traffic, measure the SLI on real requests at the load balancer, and use synthetic probes for quiet hours and for paths that users rarely take.

In an autoscaling group

An autoscaling group replaces the instances it considers unhealthy. If it takes the load balancer's health check as that signal, a failed readiness check stops meaning no traffic for now and starts meaning terminate: readiness with the consequences of liveness. A deep check that fails everywhere then makes the group replace the whole fleet. Use a separate, more conservative signal for replacement than for routing, and give new instances a grace period in which to start. Autoscaling covers the rest of that loop.

When a health check lies

A health check is a model of the service, and the model can be wrong in both directions.

• It passes while users fail. The check takes a different path from real requests: another port, no authentication, no database. Or a bad release returns errors on every real endpoint and 200 on /readyz. An instance that fails fast can even look like the least loaded one and attract more traffic.
• It fails while users are fine. The check tests something that most requests don't need, or it times out because the instance is busy doing useful work.

Catch both with signals from real traffic: each instance's error rate and latency compared with its peers, passive health checks (outlier detection) in the load balancer or the Service Mesh, alerts on the SLO's burn rate, and the external monitor. When those signals take out an instance that was passing its checks, treat the gap as a bug in the check.

When to use it

• For every service that runs behind a load balancer or under an orchestrator: at least a liveness and a readiness endpoint, kept separate even if they start out identical.
• Add a startup check when booting takes longer than the liveness check would tolerate: large caches, JIT warm-up, migrations.
• Add outside-in monitoring for anything that users or other teams depend on, and for anything that has an SLO.
• Queue consumers and batch workers have no load balancer to tell, so readiness matters little for them. Liveness still does: expose it as a tiny HTTP endpoint, or let the orchestrator run a command that checks a heartbeat.
• A health endpoint is a yes-or-no signal for automation. It does not replace metrics, logs and traces, which tell you why.

Trade-offs

• Shallow or deep. Shallow checks miss failures in dependencies; deep checks turn a dependency's failure into your own. Whatever depth you choose, know what happens when every instance fails the check at once.
• Fast or stable. Short intervals and low thresholds find failures sooner and flap more, and every probe costs something on every instance.
• Fail open or fail closed. A balancer that fails open keeps serving through a bad check, and may send traffic to instances that really are broken. One that fails closed protects users from broken instances, and turns a bad check into an outage.
• Restarts hide problems. A liveness check that restarts a leaking or deadlocking process keeps the service up and the bug alive. Count restarts and alert on them.
• One more surface. A detailed health endpoint leaks information, and one that does real work per request is a cheap target for a denial-of-service attack.

Implementation notes

Products change their defaults, so treat these as examples that were checked in October 2026, and read the current documentation.

• Kubernetes probes: startupProbe, livenessProbe and readinessProbe, run by the kubelet as an HTTP GET, a TCP connect, a command or a gRPC health call. The defaults are periodSeconds: 10, timeoutSeconds: 1, failureThreshold: 3, successThreshold: 1 and initialDelaySeconds: 0. Liveness and readiness probes do not run until the startup probe has succeeded. A failed liveness or startup probe gets the container killed and restarted under the Pod's restart policy; a failed readiness probe takes the Pod's address out of the EndpointSlices of the Services that select it. A deleted Pod gets a 30-second grace period by default.
• The Kubernetes API server follows the pattern itself with /livez and /readyz (the older /healthz has been deprecated since v1.16). Machines are meant to read only the status code; ?verbose lists the individual checks for a human, and exclude= leaves one out. With --shutdown-delay-duration set, /readyz fails as soon as shutdown begins while /livez keeps passing, which is the lame-duck sequence above.
• gRPC: the health checking protocol ships with the gRPC libraries, the kubelet's gRPC probe speaks it, and a gRPC client can be configured to watch it and stop sending calls to a backend that reports NOT_SERVING.
• Spring Boot Actuator: the health groups /actuator/health/liveness and /actuator/health/readiness report the application's availability state, and management.endpoint.health.probes.add-additional-paths=true also serves them as /livez and /readyz on the main port. DOWN and OUT_OF_SERVICE map to 503, and details are hidden by default (show-details is never).
• ASP.NET Core: MapHealthChecks exposes an endpoint. By default Healthy and Degraded return 200 and Unhealthy returns 503, the body is the status as plain text, and caching is suppressed. Tag the checks and filter on the tags to get separate liveness and readiness endpoints; RequireHost and RequireAuthorization restrict who can call them.
• When every target is unhealthy: AWS Application and Network Load Balancers fail open and route to all targets. Envoy ignores health once the healthy share of a cluster drops below its panic threshold (50% by default), or can be configured to fail the traffic instead. HAProxy answers 503 when no server is available, and Azure Load Balancer sends no new flows to a pool whose instances are all probed down.
• Replacement in a group: Amazon EC2 Auto Scaling replaces instances that fail the built-in EC2 status checks and, once a grace period has passed, any optional checks you enabled, such as the load balancer's. Google Cloud recommends a separate, more conservative health check for autohealing a managed instance group than for load balancing.
• Outside-in services: Google Cloud Monitoring uptime checks (at least three checkers) and Azure Application Insights availability tests (five or more locations recommended) both probe from several places. Alert when several locations agree, not on a single failed probe.
• Watch the health system itself: the number of ready instances against the number expected, restarts per instance, how often instances change state, and the time from a failure to its removal. An alert on fewer than N ready catches step 4 before the last instance goes.