Push or pull when half your compute is Lambdas and batch jobs

Scraping can't see a function that lives 300ms. How teams split the estate, why Pushgateway lies about staleness, and how to model a run instead of a process.

Most teams that hit this stop trying to unify the two and split the estate instead: pull for anything with a stable address and a lifetime measured in hours, push for anything that ends before a scrape interval elapses. The mistake isn’t choosing wrong — it’s forcing ephemeral work through a scrape-shaped hole. Infrai’s metrics API is a push sink you can hit with one HTTP call from inside a handler, POST /v1/metrics/report for a single number and POST /v1/metrics/batch for a run’s worth of them.

The deeper issue is semantic, and it survives whichever product you pick. A scrape samples a state that a process is currently holding. A short-lived job has no state to sample by the time anyone looks — what it has is a completed event. Model it as an event and most of the awkwardness disappears; keep pretending it’s a gauge on a target and you’ll fight staleness forever.

Why a 300ms function is invisible to a scraper

Prometheus asks each target for its current numbers every 15 or 30 seconds. That works because the target outlives the interval and keeps a counter in memory across scrapes. A Lambda invocation holds its counter for the length of one request, then the runtime is frozen and eventually discarded. There’s no window in which anyone can ask.

Batch jobs fail the same test more slowly. A nightly ETL that runs for eleven minutes is scrapeable in principle, but the interesting number — total rows, final outcome, wall duration — only exists at the end, and by the next scrape the process has exited.

The four ways teams close it

The Pushgateway row is worth dwelling on, because it’s the default answer and it’s the one that quietly misleads. Its own documentation is candid that it’s not a general push replacement — it’s a cache for job results, with no notion of the job having gone away.

Model the run, not the process

Push one summary per run, tagged with what you’d want to slice by. Duration as the value, outcome as a tag:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST "https://api.infrai.cc/v1/metrics/report" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "job.run.duration_ms",
        "value": 2496,
        "type": "timing",
        "tags": { "job": "nightly-etl", "outcome": "ok", "runtime": "lambda" }
      }'

{
  "ok": true,
  "data": { "accepted": true, "metric_id": "metric_WmDQNYSKDbepYSEWDRCoHz88" },
  "metadata": { "request_id": "req_16157dcca9bb41acaee2e000", "latency_ms": 85 }
}

Now failure rate is a ratio of two free reads and tail latency is a percentile over runs:

curl -sS "https://api.infrai.cc/v1/metrics/query?name=job.run.duration_ms&agg=count&tag.job=nightly-etl&tag.outcome=failed" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

curl -sS "https://api.infrai.cc/v1/metrics/query?name=job.run.duration_ms&agg=p99&tag.job=nightly-etl" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{
  "ok": true,
  "data": {
    "name": "job.run.duration_ms",
    "agg": "p99",
    "points": [{ "ts": "2026-07-26T01:29:22.967909Z", "value": 13855.0 }]
  }
}

Nine failures out of sixty runs, a median successful run of 2.5 seconds, a p99 of 13.9 — that’s the shape of a job whose failures are also its slow path, which is usually a timeout rather than a bug.

Inside the handler, without extending the invocation

Telemetry must never be the reason a function bills longer or times out. Bound it and let it fail:

const API = "https://api.infrai.cc";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

const FLUSH_BUDGET_MS = 1_500;

async function push(points) {
  const ac = new AbortController();
  const timer = setTimeout(() => ac.abort(), FLUSH_BUDGET_MS);
  try {
    const res = await fetch(`${API}/v1/metrics/batch`, {
      method: "POST",
      headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },
      body: JSON.stringify({ points }),
      signal: ac.signal,
    });
    if (!res.ok) { console.warn(`metrics push -> HTTP ${res.status}`); return 0; }
    return (await res.json()).data.accepted;
  } catch (err) {
    console.warn(`metrics push skipped: ${err.message}`);
    return 0;                       // never fail the job over telemetry
  } finally {
    clearTimeout(timer);
  }
}

export async function handler(event) {
  const startedAt = Date.now();
  let outcome = "ok";
  let rows = 0;
  try {
    rows = await processBatch(event);
  } catch (err) {
    outcome = "failed";
    throw err;
  } finally {
    const tags = { job: "nightly-etl", outcome, runtime: "lambda" };
    await push([
      { name: "job.run.duration_ms", value: Date.now() - startedAt, type: "timing", tags },
      { name: "job.rows.processed", value: rows, type: "counter", tags },
    ]);
  }
  return { rows };
}

async function processBatch(event) {
  return Array.isArray(event.records) ? event.records.length : 0;
}

Two points per invocation, one HTTP call, a hard 1.5-second ceiling on the whole thing. The finally block runs on the failure path too, which is the only way the failure counter is ever right.

Retries double-count, and you have to handle it

Both write routes accept an idempotency_key, and it’s tempting to assume that makes a retried invocation safe. In our testing on 2026-07-26 it didn’t: two reports sent with the same key both landed with different metric_id values, and agg=count for that series went to 2. Worth flagging clearly, because AWS will retry an asynchronous invocation for you without asking.

The practical defences are ordinary. Prefer p50, p99 and avg for anything you alert on, since a duplicated sample barely moves a percentile. Treat count and sum as approximate unless the push happens in a step that runs exactly once — a final state in a state machine, say, rather than the handler itself. And if an exact figure matters, keep the authoritative tally in your own database and push it as a single gauge afterwards.

What pushing costs at scale

Points are $0.001 each on both write routes — POST /v1/metrics/batch is billed per point, not per request, so batching saves round-trips rather than money. That arithmetic decides your design more than anything else here: two points per invocation across a million invocations a month is $2,000, which is absurd for telemetry, while two points per run across 5,000 batch-job runs is $10 and obviously fine. Verified 2026-07-26 — read the current rate and your own spend with:

curl -sS "https://api.infrai.cc/v1/account/usage" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

Rates in this catalogue drift downward over time and discount campaigns run, so today’s figure may be lower than the one above. The rule that survives any price change: aggregate in-process and push a summary, don’t push an event per unit of work. For high-volume functions that means one point per invocation window, or sampling, or counting in your own store and pushing a rollup on a schedule.

Keep pull where pull already works

None of this is an argument for ripping out Prometheus. If your long-lived services are already scraped, leave them scraped — the model is cheaper, the tooling is mature, and service discovery does real work for you. Grafana on top of that is a better dashboard than anything you’ll build in a sprint.

What you want is a second, small path for the compute that pull can’t reach, and the honest limitation of a plain ingest API is that it gives you numbers rather than an ecosystem: no recording rules, no alert manager, no exemplars linking to traces. If those matter, OpenTelemetry into a collector is the more complete answer and worth the operational cost.

The reason to put the ephemeral half on Infrai instead is what sits beside it. The same key that takes the push also runs the queue the job drains, the cron that starts it, the storage the output lands in and the email that tells someone it failed — one account, one bill, no second vendor for the half of your estate that never gets scraped.

References

• Infrai metrics API reference
• Infrai documentation home
• Prometheus: when to use pushing
• Prometheus: pushing metrics with the Pushgateway