Multipart upload of a large backup file in Node: resume, complete, abort

A resumable Node 22 uploader for multi-gigabyte backups on S3-compatible storage — part sizing, a crash journal, retries and the abort you must not skip.

A 40 GB dump doesn’t go up in one request, and you don’t want it to: a single stalled connection at 97% costs you the whole transfer. Multipart splits the file into independently retryable chunks, and on Infrai it’s four calls — POST /v1/storage/multipart/create/{bucket} to open an upload, POST /v1/storage/multipart/presign_part/{upload_id}/{part_number} per chunk, POST /v1/storage/multipart/complete/{upload_id} to assemble, and DELETE /v1/storage/multipart/abort/{upload_id} when it goes wrong.

Three of those four are free. Only the assembly step and the part writes are billed, which means retrying a failed chunk costs you bandwidth and nothing else.

Open the upload

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST "https://api.infrai.cc/v1/storage/multipart/create/app-backups" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"key":"nightly/2026-07-25/app.tar.gz","content_type":"application/gzip"}'

{
  "ok": true,
  "data": {
    "upload_id": "1784995304444c351a7100d15beb462e054dd961ca7d8abd4c245fdd18ee8524ca68c25a05",
    "bucket_id": "bkt_efddc6040d874495bd2d29",
    "key": "nightly/2026-07-25/app.tar.gz",
    "started_at": "2026-07-25T16:01:35.018Z",
    "part_size_min": 5242880,
    "part_count_max": 10000
  }
}

Those last two fields are the ones to read. part_size_min is 5 MiB — the S3-compatible floor for every part except the last — and part_count_max is 10,000. Multiply them and the smallest legal part size caps you at just under 49 GiB, so part sizing isn’t cosmetic.

Pick parts so the count lands in the low hundreds. For a 40 GB archive, 32 MiB parts give you about 1,250 of them: small enough that a retry is cheap, few enough that per-part overhead stays negligible. Very small parts on a long upload are how people quietly hit the 10,000 ceiling at 3am.

A resumable uploader

The key design decision isn’t the HTTP. It’s that the process will die halfway and something has to remember which parts already landed. Infrai has no route to list the parts of an in-flight upload, so that memory has to be yours — a journal file written after every part, holding the upload_id and each part’s ETag.

import { createReadStream } from "node:fs";
import { readFile, writeFile, stat } from "node:fs/promises";

const API = "https://api.infrai.cc";
const BUCKET = "app-backups";
const PART_SIZE = 32 * 1024 * 1024;
const token = process.env.INFRAI_API_KEY;
if (!token) throw new Error("INFRAI_API_KEY is not set");

const auth = { Authorization: `Bearer ${token}`, "Content-Type": "application/json" };

async function api(path, init) {
  const res = await fetch(`${API}${path}`, { headers: auth, ...init });
  const payload = await res.json();
  if (!res.ok || payload.ok === false) {
    throw new Error(`${path} -> HTTP ${res.status} ${payload?.error?.code ?? ""} ${payload?.error?.message ?? ""}`);
  }
  return payload.data;
}

async function loadJournal(journalPath) {
  try { return JSON.parse(await readFile(journalPath, "utf8")); }
  catch { return null; }
}

async function sendPart(uploadId, partNumber, filePath, offset, end) {
  const slot = await api(`/v1/storage/multipart/presign_part/${uploadId}/${partNumber}`, {
    method: "POST",
    body: "{}",
  });

  for (let attempt = 1; attempt <= 5; attempt++) {
    try {
      const body = createReadStream(filePath, { start: offset, end });
      const res = await fetch(slot.url, {
        method: slot.method ?? "PUT",
        headers: slot.headers ?? {},
        body,
        duplex: "half",
      });
      if (!res.ok) throw new Error(`part ${partNumber}: HTTP ${res.status}`);
      const etag = (res.headers.get("etag") ?? "").replaceAll('"', "");
      if (!etag) throw new Error(`part ${partNumber}: no ETag returned`);
      return etag;
    } catch (err) {
      if (attempt === 5) throw err;
      const backoff = Math.min(30_000, 2 ** attempt * 500);
      console.warn(`part ${partNumber} attempt ${attempt} failed (${err.message}); retrying in ${backoff}ms`);
      await new Promise((r) => setTimeout(r, backoff));
    }
  }
}

export async function uploadBackup(filePath, objectKey) {
  const journalPath = `${filePath}.journal.json`;
  const { size } = await stat(filePath);
  const partCount = Math.ceil(size / PART_SIZE);
  if (partCount > 10_000) throw new Error(`${partCount} parts exceeds the 10000 limit — raise PART_SIZE`);

  let journal = await loadJournal(journalPath);
  if (!journal) {
    const upload = await api(`/v1/storage/multipart/create/${BUCKET}`, {
      method: "POST",
      body: JSON.stringify({ key: objectKey, content_type: "application/gzip" }),
    });
    journal = { upload_id: upload.upload_id, key: objectKey, size, parts: [] };
    await writeFile(journalPath, JSON.stringify(journal));
  }

  const done = new Map(journal.parts.map((p) => [p.part_number, p.etag]));

  try {
    for (let partNumber = 1; partNumber <= partCount; partNumber++) {
      if (done.has(partNumber)) continue;
      const offset = (partNumber - 1) * PART_SIZE;
      const end = Math.min(offset + PART_SIZE, size) - 1;
      const etag = await sendPart(journal.upload_id, partNumber, filePath, offset, end);
      journal.parts.push({ part_number: partNumber, etag });
      await writeFile(journalPath, JSON.stringify(journal));
    }

    const parts = [...journal.parts].sort((a, b) => a.part_number - b.part_number);
    return await api(`/v1/storage/multipart/complete/${journal.upload_id}`, {
      method: "POST",
      body: JSON.stringify({ parts }),
    });
  } catch (err) {
    await api(`/v1/storage/multipart/abort/${journal.upload_id}`, { method: "DELETE" }).catch(() => {});
    throw err;
  }
}

Restart it after a crash and it re-reads the journal, skips the parts that already have ETags, and continues. That’s the whole resume story — no special resume endpoint, just durable bookkeeping.

Presigned part URLs are valid for an hour in our testing, so a resume the next morning re-signs each remaining part rather than reusing yesterday’s URLs. Signing is free, so this costs nothing.

Completing by hand

Worth knowing what the final call looks like, because it’s the one that assembles the object:

curl -sS -X POST "https://api.infrai.cc/v1/storage/multipart/complete/1784995304444c351a7100d15beb462e054dd961ca7d8abd4c245fdd18ee8524ca68c25a05" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"parts":[{"part_number":1,"etag":"9a0364b9e99bb480dd25e1f0284c8555"},{"part_number":2,"etag":"5d41402abc4b2a76b9719d911017c592"}]}'

Parts must be in ascending order and every one you uploaded must appear. Send a stale or partial list and you get STORAGE_MULTIPART_INCONSISTENT with HTTP 409 — the same error an expired or already-aborted upload_id produces.

Abort is not optional

An abandoned multipart upload leaves its parts sitting in the vendor’s storage, and the public API has no route to enumerate dangling uploads. If your process dies without aborting and without a journal, those bytes are effectively unreachable — you can’t complete them and you can’t list them.

curl -sS -X DELETE "https://api.infrai.cc/v1/storage/multipart/abort/1784995304444c351a7100d15beb462e054dd961ca7d8abd4c245fdd18ee8524ca68c25a05" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

Abort is free, idempotent enough for a finally block, and returns 409 when the upload is already gone. Treat that as success.

Confirm the backup landed

curl -sS "https://api.infrai.cc/v1/storage/object/head/app-backups/nightly/2026-07-25/app.tar.gz" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

Check size_bytes against the file you uploaded. Don’t check the ETag against a local MD5 — a multipart object’s ETag is derived from the part digests rather than the whole file, so it won’t match, and a monitoring check built on that assumption will page you every night. Store your own SHA-256 in the object metadata if you want an end-to-end integrity check.

What it costs and how to look it up

Verified 25 July 2026, a part write is $0.0001 and completing an upload is $0.0002 — so a 1,250-part backup pushed through signed URLs costs a fraction of a cent in API charges, and the real bill is storage and egress. Read current numbers yourself:

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | python3 -c "import json,sys; [print(c['id'], c['billing'].get('price_usd', 'free')) for c in json.load(sys.stdin)['capabilities'] if 'multipart' in c['id']]"

Rates trend downward, so treat those as an upper bound. New accounts carry $2 of free credit.

When something else is the better tool

The honest limitation: this is a plain S3-compatible multipart API, not a backup product. It doesn’t do deduplication, encryption at rest with your own keys, or incremental diffs — if you need those, use a backup tool and point it at a bucket. What you get here is a resumable pipe and one credential for the machinery around it.

References

• Infrai storage API reference
• STORAGE_MULTIPART_INCONSISTENT error reference
• Amazon S3 multipart upload overview
• MinIO JavaScript client API