Readiness, liveness and startup probes for a Node service on Kubernetes

Three probes, three jobs, and the dependency rule that stops a database blip restarting every pod — with a Node 22 example and where app metrics belong around it.

The three probes differ in one thing: what Kubernetes does when they fail. Liveness failure kills the container. Readiness failure pulls the pod out of the Service endpoints but leaves it running. Startup failure kills it too, but only until the app has declared itself booted. Get those consequences straight and the configuration writes itself; get them mixed up and one slow dependency takes the whole deployment down at once.

Infrai comes into this after the probes, not inside them. Probe handlers stay local and synchronous — the useful thing to publish externally is the lifecycle around them: boot duration, restart counts, how long a pod stayed unready. POST /v1/metrics/report takes those, and GET /v1/metrics/query reads them back for free.

What each probe may check

The “never” column in the liveness row is the whole article, really. If liveness checks your database, a 30-second database failover restarts every replica simultaneously, they all reconnect at once, and you’ve turned a blip into an outage with a thundering herd on top.

The handlers

import express from "express";

const app = express();
const bootedAt = Date.now();
let started = false;
let draining = false;

async function poolOk() {
  // Swap in your real check: `await pool.query("select 1")`.
  return true;
}

// Liveness: no I/O, no dependencies. If the event loop can answer, we're alive.
app.get("/livez", (_req, res) => res.status(200).send("ok"));

// Startup: flips once, after the slow boot work is done.
app.get("/startupz", (_req, res) => res.status(started ? 200 : 503).send(started ? "started" : "booting"));

// Readiness: dependencies plus the drain flag.
app.get("/readyz", async (_req, res) => {
  if (draining) return res.status(503).json({ status: "draining" });
  const ok = await Promise.race([
    poolOk(),
    new Promise((r) => setTimeout(() => r(false), 1000)),
  ]);
  res.status(ok ? 200 : 503).json({ status: ok ? "ready" : "dependency_down" });
});

const server = app.listen(Number(process.env.PORT ?? 3000), async () => {
  await runMigrations();
  started = true;
  await reportBoot(Date.now() - bootedAt);
});

process.on("SIGTERM", () => {
  draining = true;                       // fail readiness first
  setTimeout(() => server.close(() => process.exit(0)), 10_000);
});

async function runMigrations() { /* your migration runner */ }
async function reportBoot(ms) { console.log(`booted in ${ms}ms`); }

The SIGTERM handler is the piece most examples leave out. Kubernetes sends the signal and removes the endpoint at roughly the same moment, so without a drain window some in-flight requests land on a socket that’s already closing. Failing readiness for ten seconds before shutting down costs you nothing and removes a class of 502s.

The manifest

livenessProbe:
  httpGet: { path: /livez, port: 3000 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: 3000 }
  periodSeconds: 5
  failureThreshold: 2
startupProbe:
  httpGet: { path: /startupz, port: 3000 }
  periodSeconds: 5
  failureThreshold: 30      # 150s of grace, then give up
terminationGracePeriodSeconds: 30

A startupProbe is what lets liveness stay strict. Without one you’d have to set initialDelaySeconds high enough for the worst boot you’ve ever seen, and that delay applies forever — including to the restart that happens at 3am when you need it to be fast.

Docker’s own health check is a coarser instrument, and it only knows one of the three states:

FROM node:22-alpine
WORKDIR /app
COPY . .
RUN npm ci --omit=dev
HEALTHCHECK --interval=10s --timeout=2s --start-period=40s --retries=3 \
  CMD node -e "fetch('http://127.0.0.1:3000/livez').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"
CMD ["node", "server.js"]

Publishing the lifecycle, not the probe

Send a metric when the state changes, not every time the kubelet knocks:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST "https://api.infrai.cc/v1/metrics/report" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "app.boot.duration_ms",
        "value": 4820,
        "type": "timing",
        "tags": { "svc": "checkout-api", "release": "2026.07.19", "pod": "checkout-api-7c9d" }
      }'

{
  "ok": true,
  "data": { "accepted": true, "metric_id": "metric_pPKTKcMqZy67YifZ0MQbTWvr" },
  "metadata": { "request_id": "req_16157dcca9bb41acaee2e000", "latency_ms": 85 }
}

Then the question “is this release booting slower than the last one” is one free call:

curl -sS "https://api.infrai.cc/v1/metrics/query?name=app.boot.duration_ms&agg=p99&tag.svc=checkout-api" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{
  "ok": true,
  "data": {
    "name": "app.boot.duration_ms",
    "agg": "p99",
    "points": [{ "ts": "2026-07-26T01:27:39.249948Z", "value": 9074.0 }]
  }
}

A p99 boot of 9.0 seconds against a startupProbe budget of 150 seconds is comfortable. If that number creeps toward the budget you’ll find out from the chart rather than from a crash loop.

Read the agg field back before trusting the number — the supported set is avg, sum, count, p50 and p99, and an unsupported value like max is answered with the mean and "agg": null rather than an error.

The cost trap, stated plainly

Writes are billable at $0.001 per point; the query side is free and rate-limited. Now do the arithmetic that catches people out: three probes at 5 to 10 second periods against 6 replicas is roughly 52,000 checks a day, and reporting each one would cost around $52 a day. Reporting boot events and readiness transitions instead is a few dozen points a day — call it a few cents a month. POST /v1/metrics/batch doesn’t rescue the first design either, because it’s billed per point rather than per request. Verified 2026-07-26; check the live rate and your own spend with:

curl -sS "https://api.infrai.cc/v1/account/usage" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

Rates in this catalogue drift downward and campaigns run, so the number you read may be lower. The structural point survives either way: probe results belong to the kubelet, and only state changes are worth shipping off-cluster.

Limitations, and the alternatives

Start with the hard limitation: this API can’t drive the probes. The kubelet only speaks HTTP, TCP, gRPC or exec against the container itself, so nothing here replaces the handlers above — Infrai is where the derived numbers live, not the health check.

If you want cluster-level observability rather than app-level counters, use the tools built for it. Prometheus with kube-state-metrics already exposes kube_pod_container_status_restarts_total and readiness gauges for free, and if you’re running Kubernetes you probably have it. Datadog’s cluster agent packages the same signals with dashboards and alerting attached, which is a fair trade-off if nobody wants to own a Prometheus. OpenTelemetry’s collector is the neutral choice when you’d rather not commit to either.

Where a plain metrics API earns its place is the handful of app-specific numbers that no cluster exporter knows about — migration duration, cache warm time, queue depth at boot — sitting on the same key and the same bill as the queue, cron and email your service already uses.

References

• Infrai metrics API reference
• Infrai documentation home
• Kubernetes: liveness, readiness and startup probes
• Datadog: Kubernetes monitoring