Scraping a page to clean Markdown for RAG ingestion

One call returns readable text instead of HTML. The chunking that follows, and the two checks that stop a failed fetch becoming a poisoned index.

The unglamorous half of a RAG pipeline is turning a web page into text worth embedding. POST /v1/web/scrape on Infrai takes a url and a format of markdown or text and returns the readable content — no headless browser to run, no boilerplate stripping to maintain, and no second vendor, because the same key also holds the embeddings and the vector store you’re about to write into.

The scrape is one call. Everything interesting happens in what you do with the result.

The call

curl -sS -X POST "https://api.infrai.cc/v1/web/scrape" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.f5.com/glossary/api-gateway", "format": "markdown"}'

{
  "ok": true,
  "data": {
    "url": "https://www.f5.com/glossary/api-gateway",
    "title": "What Is an API Gateway? A Quick Learn Guide | F5",
    "content": "An API gateway accepts API requests from a client, processes them based on defined policies...",
    "format": "markdown",
    "raw_vendor_payload": null
  }
}

markdown keeps headings and lists, which matters more than it looks: a chunker that can see ## boundaries splits on meaning rather than on character count. Use text only when your downstream tooling actively dislikes markup.

title is worth storing as metadata — it’s the best short description of the chunk’s source you’ll get for free.

The two checks before you index

This is where pipelines quietly rot. A scrape that returns almost nothing — a paywall, a JavaScript-only page, a consent wall — still returns successfully, and an empty chunk embedded into your index is a row that will match queries and answer nothing.

import os
import re

import requests

API = "https://api.infrai.cc"
KEY = os.environ["INFRAI_API_KEY"]
COLLECTION = os.environ.get("RAG_COLLECTION", "docs")
MIN_CONTENT_CHARS = 500
SESSION = requests.Session()
SESSION.headers.update({"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"})


def scrape(url: str, fmt: str = "markdown") -> dict:
    resp = SESSION.post(f"{API}/v1/web/scrape", json={"url": url, "format": fmt}, timeout=60)
    body = resp.json()
    if not body.get("ok"):
        raise RuntimeError(body["error"]["code"])
    return body["data"]


def usable(page: dict) -> tuple[bool, str]:
    """Two checks that stop a bad fetch reaching the index. Neither is clever; both
    catch the failures that otherwise show up as an assistant confidently citing a
    cookie banner."""
    content = (page.get("content") or "").strip()
    if len(content) < MIN_CONTENT_CHARS:
        return False, f"only {len(content)} chars — likely a wall or a JS-only page"
    lowered = content.lower()
    if any(marker in lowered[:400] for marker in
           ("enable javascript", "accept cookies", "verify you are human", "subscribe to read")):
        return False, "content looks like an interstitial, not the article"
    return True, "ok"


def chunk(markdown: str, target_chars: int = 1200, overlap: int = 150) -> list[str]:
    """Split on heading boundaries first, then by size. Splitting purely by length
    cuts sentences in half and produces chunks that embed badly."""
    sections = re.split(r"\n(?=#{1,3}\s)", markdown)
    chunks: list[str] = []
    for section in sections:
        section = section.strip()
        if not section:
            continue
        if len(section) <= target_chars:
            chunks.append(section)
            continue
        start = 0
        while start < len(section):
            chunks.append(section[start:start + target_chars])
            start += target_chars - overlap
    return chunks


def ingest(url: str) -> dict:
    page = scrape(url)
    ok, why = usable(page)
    if not ok:
        return {"url": url, "indexed": 0, "skipped": why}

    pieces = chunk(page["content"])
    embedded = SESSION.post(f"{API}/v1/embeddings",
                            json={"model": "auto", "input": pieces}, timeout=180)
    embedded.raise_for_status()
    embeddings = [row["embedding"] for row in embedded.json()["data"]]

    vectors = [{"id": f"{url}#{i}", "embedding": embedding,
                "metadata": {"url": url, "title": page.get("title"), "chunk": i, "text": piece}}
               for i, (piece, embedding) in enumerate(zip(pieces, embeddings))]
    resp = SESSION.post(f"{API}/v1/vector/upsert",
                        json={"collection": COLLECTION, "vectors": vectors}, timeout=120)
    resp.raise_for_status()
    return {"url": url, "indexed": len(vectors), "title": page.get("title")}


if __name__ == "__main__":
    print(ingest("https://www.f5.com/glossary/api-gateway"))

Both checks are crude and both earn their place. The length floor catches paywalls and JavaScript-only pages; the interstitial check catches consent walls that are long enough to pass the floor.

Chunk on structure, not on length

Splitting every 1,200 characters is the default everyone starts with and it cuts sentences in half. Markdown gives you headings, so split on those first and only fall back to length within an oversized section — which is what the chunk function above does, and it’s most of the quality difference between a pipeline that answers well and one that returns fragments.

Keep a small overlap so a fact spanning a boundary survives in one piece.

Re-crawling without duplicating

Deterministic chunk ids — url#index, as above — mean a re-scrape overwrites the previous version of each chunk instead of adding a second copy. Without that, your index accumulates every version of every page you ever fetched, and retrieval starts returning three slightly different answers to the same question.

Schedule re-crawls on POST /v1/cron/create and let the ids do the deduplication.

Limitations

There’s no crawl: one call, one URL. Following links, respecting a sitemap, discovering new pages and scheduling depth-limited traversal are all yours to build, and if that’s the job you have, Firecrawl’s crawl pipeline exists precisely for it and will save you weeks. Exa is the one to look at if semantic retrieval over the web is closer to your problem than fetch-and-chunk.

Scraping is also Tavily-backed here, so the fetch behaviour, the rendering and the boilerplate stripping are theirs — using Tavily directly gets you their full extraction options, including their own answer synthesis. And nothing in this surface handles authentication walls: a page behind a login isn’t a good fit, and neither is one that only assembles itself after several seconds of client-side rendering.

What one credential gives you is the whole pipeline without a second account: scrape, then POST /v1/embeddings on the OpenAI-compatible surface for vectors, then POST /v1/vector/upsert to store them, then POST /v1/vector/query at answer time, with POST /v1/cron/create driving the refresh and one GET /v1/account/usage pricing all of it. Scrape bills per call, live in GET /v1/discovery/web.scrape (verified 2026-09-21), and platform rates drift downward as vendor contracts improve.

References

• Infrai web API reference
• Infrai vector API reference
• Infrai documentation home