Nightly data cleanup in Node: a cron-hit endpoint that fans out to a queue

The scheduled endpoint should enumerate, not delete. A worked Node 22 example: a secret-protected HTTP handler, bounded batches on a queue, and a worker that can run for an hour.

The nightly cleanup that works at 10,000 rows and falls over at 4 million has one design flaw: the scheduled HTTP handler does the deleting. Split it. Let the handler enumerate what’s stale and hand out bounded chunks of work, then let a process that isn’t tied to a request lifetime chew through them. Infrai’s queue is the handoff — a publish per chunk, free consumes, and a dead-letter lane for the chunk that keeps blowing up.

Whatever fires the endpoint barely matters. What matters is that the endpoint returns in a second.

Pick the simplest thing that survives your volume

If your table is partitioned by month and you only ever drop whole months, stop reading and go do that — no queue is faster than dropping a partition. The fan-out below earns its keep when cleanup spans more than one system: rows in Postgres, exported CSVs in object storage, a search index entry, maybe a cache key.

Create the queue

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST "https://api.infrai.cc/v1/queue/create" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"name":"nightly-purge","type":"standard","dlq":"nightly-purge-dlq"}'

Messages are capped at 256 KB and retained for 14 days by default, so chunks travel as ID lists, never as row contents. A payload that carries whole records will eventually hit QUEUE_MESSAGE_TOO_LARGE.

The endpoint: authenticate, enumerate, return

Two mistakes are common here. The first is doing the work inline. The second is leaving the endpoint open, because a URL that deletes data is a URL somebody will find. Check a shared secret before anything else.

import { createServer } from "node:http";
import { timingSafeEqual } from "node:crypto";
import process from "node:process";
import pg from "pg";

const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
const CRON_SECRET = process.env.CRON_SECRET;
const INFRAI_KEY = process.env.INFRAI_API_KEY;
if (!CRON_SECRET || !INFRAI_KEY) throw new Error("CRON_SECRET and INFRAI_API_KEY must both be set");

const CHUNK = 1000;
const RETAIN_DAYS = 90;

function authorized(req) {
  const got = Buffer.from(req.headers["x-cron-secret"] ?? "");
  const want = Buffer.from(CRON_SECRET);
  return got.length === want.length && timingSafeEqual(got, want);
}

async function publishChunk(ids, runId) {
  const res = await fetch("https://api.infrai.cc/v1/queue/publish", {
    method: "POST",
    headers: { Authorization: `Bearer ${INFRAI_KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify({ queue: "nightly-purge", body: { run_id: runId, ids } }),
  });
  const out = await res.json();
  if (out.ok === false) throw new Error(`publish failed: ${out.error.code} ${out.error.message}`);
  return out.data.message_id;
}

createServer(async (req, res) => {
  if (req.url !== "/jobs/nightly-purge" || !authorized(req)) {
    res.writeHead(403).end("forbidden");
    return;
  }
  const runId = new Date().toISOString().slice(0, 10);
  const cutoff = new Date(Date.now() - RETAIN_DAYS * 864e5).toISOString();
  let after = "00000000-0000-0000-0000-000000000000";
  let chunks = 0;

  try {
    for (;;) {
      const { rows } = await pool.query(
        "SELECT id FROM webhook_logs WHERE created_at < $1 AND id > $2 ORDER BY id LIMIT $3",
        [cutoff, after, CHUNK],
      );
      if (rows.length === 0) break;
      await publishChunk(rows.map((r) => r.id), runId);
      after = rows[rows.length - 1].id;
      chunks += 1;
    }
    res.writeHead(202, { "content-type": "application/json" }).end(JSON.stringify({ run_id: runId, chunks }));
  } catch (err) {
    console.error(`enumeration failed after ${chunks} chunks: ${err.message}`);
    res.writeHead(500).end(JSON.stringify({ error: err.message, chunks }));
  }
}).listen(8080);

Keyset pagination (id > $2) rather than OFFSET keeps every page the same cost as the first, which matters once you’re 3,000 pages in. The handler holds no locks, touches an index only, and typically returns in well under a second for a few thousand chunks.

Firing it from a scheduler you already have

GitHub Actions is free for this and needs no infrastructure:

name: nightly-purge
on:
  schedule:
    - cron: "17 3 * * *"
  workflow_dispatch:
jobs:
  trigger:
    runs-on: ubuntu-latest
    steps:
      - name: Hit the cleanup endpoint
        run: |
          curl -sS -f -X POST "https://api.example.com/jobs/nightly-purge" \
            -H "X-Cron-Secret: ${{ secrets.CRON_SECRET }}"

Scheduled workflows can start late when the platform is busy — GitHub documents the delay — so don’t build anything that assumes 03:17 exactly. Pick an odd minute anyway; the top of the hour is where every cron on the internet piles up.

The worker, which is allowed to take an hour

import process from "node:process";
import { setTimeout as sleep } from "node:timers/promises";
import pg from "pg";

const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
const headers = {
  Authorization: `Bearer ${process.env.INFRAI_API_KEY}`,
  "Content-Type": "application/json",
};

async function api(path, payload) {
  const res = await fetch(`https://api.infrai.cc${path}`, { method: "POST", headers, body: JSON.stringify(payload) });
  const out = await res.json();
  if (out.ok === false) throw new Error(`${path}: ${out.error.code} — ${out.error.message}`);
  return out.data;
}

let running = true;
process.on("SIGTERM", () => { running = false; });

while (running) {
  const { items } = await api("/v1/queue/consume", { queue: "nightly-purge", max_messages: 10 });
  if (items.length === 0) { await sleep(10_000); continue; }

  for (const msg of items) {
    const { ids, run_id } = msg.payload;
    try {
      const del = await pool.query("DELETE FROM webhook_logs WHERE id = ANY($1)", [ids]);
      console.log(`run ${run_id}: deleted ${del.rowCount} of ${ids.length} (delivery ${msg.delivery_count})`);
      await api("/v1/queue/ack", { queue: "nightly-purge", receipt_handle: msg.message_id });
    } catch (err) {
      console.error(`chunk failed, leaving it for redelivery: ${err.message}`);
    }
  }
}

Deleting by primary key is what keeps this safe — each statement touches at most 1,000 rows and commits, so nothing holds a lock long enough to matter. A chunk that throws is deliberately not acked: the lease lapses after the visibility timeout (300 seconds by default), the message reappears with a higher delivery_count, and the third failure sends it to nightly-purge-dlq. That’s your poison-chunk detector, and it costs nothing to run.

Read the queue’s configuration back whenever you’re unsure what you created:

curl -sS "https://api.infrai.cc/v1/queue/get/nightly-purge" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

What a month of this costs

Publishing is the only metered call at $0.00002 each, verified 2026-07-26; consuming, acking and reading the dead-letter queue are free but rate-limited. A nightly run that enqueues 2,000 chunks is $0.04 a night, near enough $1.20 a month, and the retries are free — which is the ratio that matters, since cleanup jobs fail more often than they succeed cleanly. Check the real figure against your own account:

curl -sS "https://api.infrai.cc/v1/account/usage" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

That returns total_cost, total_calls and a breakdown array where queue.publish appears as its own line. Per-call rates drift downward and promotions happen, so treat the figure above as a ceiling.

Honest alternatives

BullMQ is the better fit if Redis is already running and you want repeatable jobs, concurrency controls and a dashboard in one library. Celery plays the same role for a Python codebase. SQS is right when the rest of the system is AWS and you want IAM policies rather than a bearer token. And if your cleanup is genuinely one SQL statement against one database, none of this applies — put it in a plain cron job and move on.

Where this shape wins is the multi-system cleanup: the same key that drains the queue also deletes the exported file from object storage and files the failure, on one bill. The limitation to weigh is that Infrai’s queue has no cron expression of its own, so something external still has to hit that endpoint every night.

References

• Infrai queue API reference
• QUEUE_MESSAGE_TOO_LARGE error reference
• GitHub Actions: events that trigger workflows
• PostgreSQL DELETE reference