Below is what actually goes stale, what a full rebuild really costs (less than you'd think in embeddings, more elsewhere), a do-it-yourself content-hash pattern you can drop into any scraper, and how we handle it in MeshArc, the crawler I'm building. The first half needs no sign-up for anything.

What goes wrong when a knowledge base ages

A stale chunk does not announce itself. It still sits close to the query in vector space, so retrieval still ranks it highly and the model still quotes it with confidence. Three failure modes are worth separating:

1. Stale chunks. The page changed; the index holds the old wording. The assistant recommends a deprecated flag or an old price.
2. Zombie chunks. The page was deleted, but its vectors were not. Nightly upserts never remove anything, so deleted pages live on indefinitely unless something deletes them on purpose.
3. Flicker re-embeds. A cookie banner, a "last updated 3 minutes ago" line or a rotating testimonial makes every page look modified on every run, so a "changed pages only" pipeline quietly turns back into a full rebuild.

There is a fourth, nastier one: the blocked crawl that deletes your index. If the site rate-limits you halfway through a run and your pipeline treats "not seen this run" as "removed", you wipe half the knowledge base. Any deletion logic needs a guard against partial runs.

What a full nightly rebuild costs

Here is the arithmetic for a mid-sized documentation site. The assumptions are ours and stated so you can swap in your own:

• 2,000 pages, averaging 1,200 words each.
• Chunks of about 350 words with a 40-word overlap, which gives roughly 4 chunks per page and 8,000 chunks in all.
• About 1.3 tokens per English word.
• OpenAI text-embedding-3-small at $0.02 per million input tokens (OpenAI's model page, checked on 23 October 2026).

That is roughly 2.8 million embedded words, or 3.7 million tokens, per full rebuild: about $0.07. Thirty nights is about $2.20 a month.

So, to be fair to the brute-force approach: with a small embedding model, the embedding bill is not what hurts. A larger model, a bigger corpus or a hosted embedding service with per-request pricing changes that, but the real costs of rebuilding everything are usually elsewhere:

• Fetching. 2,000 pages × 30 nights is 60,000 page reads a month, every one of which can be blocked, rate-limited or billed.
• Index writes. 8,000 chunk upserts a night, most of them writing identical vectors over identical vectors. Managed vector databases tend to price writes, and self-hosted ones still pay for them in compaction and I/O.
• Correctness. A rebuild that overwrites by ID still leaves zombies behind, and a rebuild that drops and recreates the index has a window in which the assistant answers from nothing.

The fix for all of these is the same: find out what changed, and touch only that.

The do-it-yourself version: hash, compare, delete deliberately

You do not need any particular vendor for this. The pattern is:

1. Normalise each page's text (strip boilerplate, collapse whitespace) and hash it.
2. Compare against the hash you stored last time.
3. Re-chunk and re-embed only pages whose hash moved; give every chunk an ID that starts with the page URL.
4. Before adding a changed page's new chunks, delete all of its old ones. The chunk count can shrink, so overwriting by ID is not enough.
5. Delete pages that have disappeared, but only if the run reached most of the site.

import hashlib, json, re
from pathlib import Path

STATE = Path("hashes.json")

def normalise(text: str) -> str:
    text = re.sub(r"Last updated[^\n]*", "", text)   # add your own volatile lines
    return re.sub(r"\s+", " ", text).strip()

def plan_refresh(pages: dict[str, str], expected_urls: int):
    """pages: {url: markdown} from this run. Returns what to embed and delete."""
    old = json.loads(STATE.read_text()) if STATE.exists() else {}
    new = {u: hashlib.sha256(normalise(t).encode()).hexdigest() for u, t in pages.items()}

    changed = [u for u, h in new.items() if old.get(u) != h]
    removed = [u for u in old if u not in new]

    # Guard: a run that saw under 90% of the site must not delete anything.
    if len(pages) < 0.9 * expected_urls:
        removed = []
        new = {**old, **new}          # keep unseen pages' hashes for next time

    STATE.write_text(json.dumps(new))
    return changed, removed

# changed -> delete chunks with IDs starting f"{url}#", then embed and add new chunks
# removed -> delete chunks with IDs starting f"{url}#"

Two honest caveats. First, this still fetches every page every night to compute the hashes; it saves embedding and index writes, not crawling. Second, the normalise function is where the work lives. Every site has its own flicker, and you will be adding regexes for months.

If you are already on LangChain, its indexing API (a record manager plus index() with incremental cleanup) does the hash-and-skip part for documents you hand it. It does not fetch pages or know whether a crawl was partial, so the first and last steps above remain yours.

Skipping the fetch as well: sitemap lastmod

Many sites publish a lastmod date per URL in their sitemap. If it is trustworthy, you can skip fetching pages that have not changed at all. "If" is doing real work there: plenty of sitemaps stamp every URL with today's date, and some never update it. A sensible incremental crawler checks how many entries carry lastmod, falls back to a full crawl when coverage is poor, and runs a full pass periodically anyway to catch pages whose dates lie.

How MeshArc does it

This is the part where we describe our own product, so weigh it accordingly. Everything below is from our docs and pricing page.

A project is a site plus its settings, run on a schedule (hourly, daily, weekly or manual). Every run is diffed against the previous one, page by page: added, modified, removed, same, withheld. Boilerplate such as cookie banners, newsletter prompts and share bars is stripped by default, and a modified page with fewer than notify_min_words changed words (default 5) does not count as changed.

With sitemap_incremental on, a scheduled run fetches only pages the sitemap shows as new or with a newer lastmod, and carries the rest forward from the last full run. A full run is forced every sitemap_full_every runs (default 10), whenever fewer than 90% of entries carry lastmod, or when a carried page is over 30 days old. Carried-forward pages cost 0 credits. So do refused pages, 404s and 410s.

Removals are only reported when a run reached at least 90% of the site. Below that, missing pages are withheld, not removed, so a blocked crawl cannot empty your index.

pip install mesharc langchain-mesharc

from mesharc import MeshArc

arc = MeshArc()  # reads MESHARC_API_KEY

project = arc.projects.create(
    "https://docs.example.com",
    name="Docs KB",
    schedule="daily",
    config={"include_paths": ["/docs/*"], "sitemap_incremental": True},
)

run = arc.runs.start(project["id"], wait=True)   # first run is the baseline
record = arc.changes(project["id"])
print(record["change"]["counts"])

Then, after each scheduled run (from the run.finished webhook or a cron of your own), load only what changed and apply it to any LangChain vector store:

from langchain_mesharc import MeshArcLoader

def refresh(store, project_id: str) -> None:
    docs = MeshArcLoader(mode="changes", project_id=project_id,
                         include_removed=True).load()
    gone = [d.metadata["url"] for d in docs if d.metadata["change"] == "removed"]
    fresh = [d for d in docs if d.metadata["change"] != "removed"]
    if gone:
        store.delete(ids=gone)
    if fresh:
        store.add_documents(fresh, ids=[d.metadata["url"] for d in fresh])

That keys one document per URL. If you split pages into chunks before embedding, use chunk IDs that start with the URL and delete a page's old chunks first, exactly as in the DIY version. The LangChain integration docs have the details.

If you would rather not run that code at all, MeshArc can write straight into Pinecone, Qdrant, Weaviate or pgvector as a destination. It splits Markdown into paragraph-aligned chunks of about 350 words with a 40-word overlap, embeds them with your own OpenAI-compatible endpoint and key (we add no markup on embeddings), and by default writes only pages classed as added, modified or removed. Destinations are included on every plan.

The same docs site, priced

Back to the 2,000-page site, read daily on our Growth plan ($99 for 500,000 credits, so about $0.20 per 1,000 plain fetches). Assume the pages fetch plainly at 1 credit each, and that around 2% change on a typical day — your site's real churn will differ, which is the point of measuring it.

Embeddings follow the same shape: only the ~40 changed pages a day are re-embedded, instead of all 2,000. Pages that need a real browser cost 4 credits rather than 1, so multiply accordingly; our Firecrawl pricing breakdown works through the per-rung maths in more detail.

Where this does not fit

• One-off corpora. If you index a site once for a demo, none of this matters. Any scraper will do.
• Sites without a usable sitemap. Incremental fetching needs lastmod. Without it, MeshArc still diffs and still writes only changed pages to your store, but it has to fetch every page to know.
• Logged-in content. Pages behind your own SSO are better pulled from the source system (Confluence, Notion, a CMS export) than scraped.
• Self-hosting. MeshArc is not open source. If you need the crawler on your own hardware, the DIY approach above with an open-source crawler is the honest answer; our comparison page lists the trade-offs by vendor.

Try it on your own docs

The quickest test is to point a project at a site you already index, let it run twice, and look at the change counts: that number, not the page count, is what your pipeline should be embedding each night. The RAG pipeline use case has the overview; if your assistant reads whole sites at once, the llms.txt generator is a free alternative to embeddings altogether.

Try a page in the playground · Start free — 1,000 credits, no card

Originally published on mesharc.dev.

Every Firecrawl figure below comes from firecrawl.dev/pricing and the Firecrawl billing docs, checked on 21 October 2026. Prices change; if you are reading this much later, check them again.

Disclosure: I build MeshArc, which competes with Firecrawl. I have tried to be fair, and I say plainly where Firecrawl is the cheaper choice.

The plans, converted to dollars per 1,000 credits

Firecrawl sells monthly credit allowances. Converting each plan to a cost per 1,000 credits is the first step, because a credit is a page only in the simplest case.

Two things stand out. First, Hobby is nearly four times dearer per credit than Standard, so anyone doing real volume ends up on Standard or above. Second, the annual discount is worth about 16% and is how most comparisons (including ours on /compare) arrive at "$0.83 per 1,000 pages".

Firecrawl's concurrency also scales with plan: 2 on Free, 5 on Hobby, 25 on Standard, 50 on Growth, 100 on Scale. If you need throughput rather than volume, that can push you up a tier before credits do.

What a credit buys

From the billing docs, the per-page costs that matter for most scraping jobs:

The enhanced-proxy line deserves credit. Firecrawl does not charge extra when a page needs a better proxy, and escalated retries are not billed separately. For a site that needs residential-grade routing on every page, that flat 1 credit is very hard to beat — more on that below.

The JSON line is the one that changes bills. A page with formats: ["json"] costs 5 credits, not 1. Modifiers stack, so JSON plus Zero Data Retention is 6.

The 404 question

This is the line most people miss. From Firecrawl's billing docs: if a document is returned — including a 403 or a 404 — the page costs 1 credit plus any option costs. Only when no document comes back at all (the site never responded) is it free.

The docs suggest checking metadata.statusCode to spot blocked URLs. That is good advice, but by then you have paid. On a crawl driven by an old sitemap, or a list of product URLs scraped six months ago, 404s are not rare. And if you asked for JSON, each 404 costs 5 credits.

Worked examples in dollars

All Firecrawl prices below use Standard at $0.99 per 1,000 credits (monthly) with the annual $0.83 in brackets.

1. 1,000 plain pages, Markdown. 1,000 credits → $0.99 ($0.83).

2. 1,000 product pages, extracted to JSON. 1,000 × 5 = 5,000 credits → $4.95 ($4.15). The model cost is included, which is convenient.

3. The same 1,000 JSON pages, where 80 URLs are now 404s. Still 5,000 credits → $4.95 ($4.15). The 80 dead URLs cost 400 credits between them and returned nothing useful.

4. A 50-page PDF manual. 50 credits for parsing on top of the scrape → about $0.05. Cheap; PDFs are not where Firecrawl bills hurt.

5. Re-checking 2,000 pages weekly with Monitor. 2,000 × 4 checks = 8,000 credits a month → $7.92 ($6.64), whether or not anything changed.

Expiry, rollover and top-ups

• Plan credits reset monthly and do not roll over on Free, Hobby, Standard or Growth. Annual Scale rolls unused credits over one month; annual Enterprise, two.
• Pay-as-you-go top-ups come in $5 increments: 1,000 credits on Hobby, 2,000 on Standard, 2,500 on Growth, 5,000 on Scale. On Standard that is $2.50 per 1,000 credits — 2.5× the plan rate. Top-ups stay on the account until used, but expire if you cancel the subscription.
• Pay-as-you-go needs a paid plan; on Free, requests return HTTP 402 when credits run out.

The practical effect: size your plan to your typical month, not your peak, and treat top-ups as expensive insurance.

The same jobs on MeshArc

MeshArc bills by the rung a page actually needed, and does not bill pages that did not deliver. From mesharc.dev/pricing, on Growth ($99 for 500,000 credits, so $0.198 per 1,000 credits):

Refused pages, 404s and 410s, pages served from cache, URLs blocked by robots.txt and pages carried forward unchanged on an incremental run cost nothing. Structured extraction is 1 credit per 1,000 tokens, with the model bill paid to your own provider.

Running the examples again:

• 1,000 plain pages: $0.20 if they fetch plainly, $0.79 if every one needs a browser.
• 1,000 JSON pages, 80 of them 404s: the 80 cost 0. The other 920, at about 3,000 tokens each (the example on our pricing page), are 3 credits each for extraction plus the read — but you also pay your model provider. If that model bill matters to you, Firecrawl's all-in 5 credits may well be simpler.
• A 100-page Cloudflare-walled site: one render earns a session, then 99 pages read with it: 202 credits, about $0.04.
• 2,000 pages, 4 runs a month, 20% needing a browser, 2% residential: our pricing page puts that at about 15,200 credits, or roughly $3.01. Firecrawl Monitor on the same pages is $7.92 monthly — though Monitor gives you diffs, and so does MeshArc, so compare like with like.

Where Firecrawl is cheaper

Be honest about the residential case. A page that needs a residential IP costs MeshArc 16 credits ($3.17 per 1,000 on Growth). Firecrawl charges 1 credit with enhanced proxy, so $0.83–$0.99 per 1,000. If your whole job is hard sites that need residential routing on every page and cannot be read with a session, Firecrawl is three to four times cheaper per page. Our Firecrawl alternative page also says Firecrawl is the better pick if you want to self-host an open-source crawler; MeshArc is not open source.

Firecrawl's JSON format also includes the model. If you do not want a model key of your own, that is a real convenience.

Where the maths goes the other way is plain pages (about 4× cheaper on MeshArc), walled sites that a session can read (about 2×), anything with a meaningful 404 rate, and repeat crawls where most pages have not changed.

Estimate your own bill

Here is a small, dependency-free calculator for Firecrawl's side, using the rates above. Change the inputs to your workload.

# Firecrawl monthly cost estimate (rates checked 21 Oct 2026)
PLAN_USD_PER_1K = {"standard_monthly": 0.99, "standard_annual": 0.83,
                   "growth_monthly": 0.80, "growth_annual": 0.67}

def firecrawl_cost(pages, json_share=0.0, pdf_pages=0, zdr=False,
                   plan="standard_monthly"):
    per_page = 1 + (1 if zdr else 0)
    credits = pages * per_page + pages * json_share * 4 + pdf_pages
    return credits, round(credits / 1000 * PLAN_USD_PER_1K[plan], 2)

# 10,000 pages, 30% as JSON, 200 PDF pages
print(firecrawl_cost(10_000, json_share=0.3, pdf_pages=200))
# (22200.0, 21.98)

Note that 404s are not discounted in that formula — because Firecrawl does not discount them.

To see what the same URLs cost on MeshArc, run them through the SDK and add up the per-page credits field. Every row tells you the HTTP status, the verdict and what it cost.

pip install mesharc        # or: npm install mesharc

from mesharc import MeshArc

arc = MeshArc()  # reads MESHARC_API_KEY

urls = [line.strip() for line in open("urls.txt") if line.strip()]
batch = arc.scrape(urls, config={"concurrency": 4})

total = 0
for row in batch["pages"]:
    total += row["credits"]
    print(row["url"], row["httpStatus"], row["verdict"], row["credits"])

print(f"{len(urls)} URLs, {total} credits, ~${total / 1000 * 0.198:.2f} on Growth")

A 404 or a refused page shows up in that list with credits of 0.

The short version

• Firecrawl Standard is $0.99 per 1,000 pages billed monthly, $0.83 billed annually.
• JSON extraction makes it $4.95 / $4.15 per 1,000, model included.
• 403s and 404s are billed. Plan credits do not roll over below Scale. Top-ups cost 2.5× the plan rate on Standard.
• Enhanced proxy is free on top, which makes Firecrawl the cheaper choice for hard, residential-only sites.

If you want to compare on your own URLs rather than mine, paste a few into the MeshArc playground and look at the credits column, or see the full vendor table on /compare. Re-crawling the same sites every week? The website change monitoring use case shows how unchanged pages drop to zero.

Try it in the playground · Start free — 1,000 credits, no card

Originally published on mesharc.dev.