Cloud cron or a queue when the API you're batching against is rate limited

Vercel Cron, GitHub Actions and Cloud Scheduler start a run; none of them pace it. How to meter 8,000 calls against a 60/min vendor limit, and what the run costs.

A schedule answers one question: when does the run start. It says nothing about the two hours in the middle, and that gap is exactly where a rate-limited vendor API hurts. If the ceiling is 60 requests a minute and you have 8,000 records to enrich, what you need is something that meters the work out and remembers which rows are done. Infrai’s queue does that behind the same key as the rest of the platform — publishing is billed per call, consuming and acking are free.

Cron still fires the run. It just shouldn’t be the thing grinding through the list.

A schedule is a starting gun, not a throttle

The failure mode is boring and universal. Your scheduled function wakes up, loops over 8,000 rows as fast as the event loop allows, collects 7,400 HTTP 429s, and dies at the platform’s duration ceiling with no record of which rows succeeded. Next run repeats it. The cloud schedulers people reach for all share that shape — they were built to trigger work, not to hold state about individual items.

The last row is the one that matches a rate limit. 8,000 items at 60 per minute is roughly 133 minutes of wall clock, and no serverless invocation should be alive that long — but a worker that consumes ten messages, spends a second on each, and comes back for more can run for as many hours as the vendor’s limit requires.

Set up the queue and its failure lane

Two decisions at create time: the name, and where poison messages land.

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST "https://api.infrai.cc/v1/queue/create" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"name":"enrich-batch","type":"standard","dlq":"enrich-batch-dlq"}'

The response hands back every default you didn’t set, and those defaults are the real contract:

{
  "ok": true,
  "data": {
    "name": "enrich-batch",
    "type": "standard",
    "message_retention_days": 14,
    "max_message_size_kb": 256,
    "visibility_timeout_default": 300,
    "max_receive_count": 3,
    "dlq_name": "enrich-batch-dlq"
  }
}

Read visibility_timeout_default carefully — 300 seconds is how long a consumed message stays invisible before it comes back. For paced work that’s a feature: a worker that dies mid-batch loses at most five minutes of progress on ten items.

Load the batch

Publishing is one call per message, so this is the part that costs money. Keep payloads to an identifier plus whatever the worker can’t cheaply look up.

import process from "node:process";

const BASE = "https://api.infrai.cc";
const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("INFRAI_API_KEY is not set");
const headers = { Authorization: `Bearer ${key}`, "Content-Type": "application/json" };

async function post(path, payload) {
  const res = await fetch(`${BASE}${path}`, { method: "POST", headers, body: JSON.stringify(payload) });
  const out = await res.json();
  if (out.ok === false) throw new Error(`${path}: ${out.error.code} — ${out.error.message}`);
  return out.data;
}

export async function enqueueRows(rows) {
  let queued = 0;
  for (const row of rows) {
    const { message_id } = await post("/v1/queue/publish", {
      queue: "enrich-batch",
      body: { row_id: row.id, domain: row.domain },
    });
    if (message_id) queued += 1;
  }
  console.log(`queued ${queued} of ${rows.length}`);
  return queued;
}

One caveat we hit while testing: publishing to a queue name that doesn’t exist doesn’t fail. The API creates a standard queue on the spot, so a typo in queue silently sends 8,000 messages somewhere nobody consumes. Spell it once, in a constant.

Drain at exactly the rate the vendor allows

The worker is where the rate limit lives — not in the scheduler, not in the publisher.

import process from "node:process";
import { setTimeout as sleep } from "node:timers/promises";

const BASE = "https://api.infrai.cc";
const headers = {
  Authorization: `Bearer ${process.env.INFRAI_API_KEY}`,
  "Content-Type": "application/json",
};
const MIN_INTERVAL_MS = 1000; // 60 vendor calls per minute

async function post(path, payload) {
  const res = await fetch(`${BASE}${path}`, { method: "POST", headers, body: JSON.stringify(payload) });
  const out = await res.json();
  if (out.ok === false) throw new Error(`${path}: ${out.error.code} — ${out.error.message}`);
  return out.data;
}

export async function drain(enrich) {
  let idle = 0;
  while (idle < 3) {
    const { items } = await post("/v1/queue/consume", { queue: "enrich-batch", max_messages: 10 });
    if (!items.length) { idle += 1; await sleep(5000); continue; }
    idle = 0;
    for (const msg of items) {
      const started = Date.now();
      try {
        await enrich(msg.payload);
        await post("/v1/queue/ack", { queue: "enrich-batch", receipt_handle: msg.message_id });
      } catch (err) {
        console.warn(`row ${msg.payload.row_id} attempt ${msg.delivery_count} failed: ${err.message}`);
      }
      const spent = Date.now() - started;
      if (spent < MIN_INTERVAL_MS) await sleep(MIN_INTERVAL_MS - spent);
    }
  }
}

Three details do the real work here. max_messages is capped at 10 per call — ask for 100 and you get an INVALID_ARGUMENT — so the loop shape above isn’t optional, it’s the API’s shape. The pacing gate sits after the vendor call, which means a slow vendor response counts toward your interval instead of stacking on top of it. And a failed row is simply not acked: the lease expires, the message returns, delivery_count goes up, and after the third delivery it lands in the dead-letter queue by itself. In our testing on 2026-07-26 that transition took exactly three deliveries, with the failed message keeping its original message_id.

Check the wiring without guessing:

curl -sS "https://api.infrai.cc/v1/queue/get/enrich-batch" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

GET /v1/queue/stats/{queue} reports live counts if you want a progress bar; the dead-letter queue is an ordinary queue, so POST /v1/queue/consume on enrich-batch-dlq shows you the rows that never made it.

What the run costs

Only POST /v1/queue/publish is billable: $0.00002 per message, verified 2026-07-26. Create, consume, ack and the DLQ routes are free and rate-limited rather than metered, which is the part that matters for a paced drain — redelivery of a failed row costs nothing, so a retry storm burns time, not budget. 8,000 rows is $0.16. New accounts get $2 of credit, roughly 99,999 publishes.

Get today’s number rather than trusting this page:

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '.capabilities[] | select(.id | startswith("queue.")) | {id, price: .billing.price_usd, free: .billing.free}'

Rates here move down over time and discount campaigns run, so what that prints may well be lower than $0.00002. The durable argument isn’t the rate anyway — it’s that the enrichment result you’re about to store, the failure email you’ll send, and the per-tenant cost attribution you’ll be asked for at the end of the quarter are all on this same key and this same bill.

When the scheduler alone is the right call

If the batch is 200 rows against a vendor that allows 5 requests per second, a plain loop inside a scheduled function finishes in under a minute and needs none of this. Stick with cron and skip the queue.

Past that, name your constraint honestly. SQS is the better pick if you’re already AWS-shaped and want IAM doing the access control; QStash is purpose-built if the work is nothing but outbound HTTP on a delay; BullMQ wins when Redis is already running and you want job classes in-process. Infrai’s queue earns the slot when you’d otherwise stand up a broker plus a worker framework for a few thousand jobs a day.

The limitations are worth knowing first: ten messages per consume call, no built-in rate limiter (the pacing above is yours to write), and no cron expressions inside the queue itself — something outside still has to start the worker.

References

• Infrai queue API reference
• Vercel: cron job usage and pricing
• Amazon SQS dead-letter queues