David's Archive
HomeLibraryStatsAbout
Log in

About this project

This product compresses a lot of what I read and listen to online into a knowledge base that a user can interact with. Every post here is scraped, summarized, chunked, and embedded.

Feel free to ask questions, and the archive will retrieve the most relevant passages and stream back a cited answer. For myself, there is a spaced repetition loop that quizzes me on what I've saved, so the archive compounds instead of just accumulating.

Methodology

Curation — what gets in

  • What earns a save

    The bar is not whether something was interesting to read, it is whether I expect to want it back when I am deciding something months from now. Most of what I open never makes it in, and the archive is more useful for what it excludes than for what it holds.

    Browse what made the cut
  • The note is the point

    Anything I keep gets my own commentary attached, written as an argument rather than a highlight, and that commentary is chunked and embedded alongside the article text. The archive can then answer with what I concluded, not only with what the author claimed.

    Ask the archive

Answering — how a question gets resolved

  • Retrieval before generation

    A question is embedded and matched against passages rather than handed straight to a model, so answers come from the corpus instead of from whatever the model happens to remember. When nothing relevant comes back, the honest answer is that the archive does not cover it.

    Read the pipeline
  • Every answer shows its work

    Responses stream with citations that link back to the exact post they came from, and the panel behind each answer lists the passages retrieved along with their similarity scores. You can check the reasoning rather than trust it.

    See a cited answer

Building — how it stays honest

  • Eval-driven, not vibes-driven

    A golden set of questions scores retrieval hit rate, citation accuracy, and answer quality on a weekly cron, and the results get published whether or not they flatter the system. Changes to the pipeline are judged against that scorecard instead of against my impression of them.

    Read the scorecard
  • Built to be measured

    Instrumented events roll up nightly into a public stats page that states its own metric framework, north star, activation, and retention proxy, in plain prose. The numbers are visible for the same reason the citations are.

    See the numbers

How it's built

The pipeline

  1. 1Capture

    URL, PWA share, or bookmarklet

  2. 2Enrich

    scrape + Gemini summary & tags

  3. 3Chunk & embed

    body + curator notes → 768-dim vectors

  4. 4Retrieve

    Firestore findNearest + curator boost

  5. 5Answer

    streamed, with [^slug] citations

  6. 6Measure

    events, rollups, weekly evals

Stack

Next.js 15 (App Router, ISR, server actions) · TypeScript · Firebase Firestore with native vector search · Gemini for enrichment, embeddings, generation, and eval judging · Tailwind + shadcn/ui · Vercel with cron-driven rollups, digests, and eval runs.

Tradeoffs

Firestore vector search over a dedicated vector DB

The corpus is thousands of chunks, not millions. findNearest keeps retrieval inside the primary datastore — one consistency model, one backup story, zero extra infrastructure — at the cost of fewer tuning knobs. The right call until scale says otherwise.

Live published-only filtering over precomputed snapshots

Retrieval re-checks post status at query time instead of maintaining a denormalized public corpus. Slightly more reads per query, but drafts can never leak through a stale snapshot — a security property worth paying latency for.

Firestore-backed rate limiting over Redis

A sliding-window counter in a ratelimits collection is slower and coarser than Redis, but adds no infrastructure to a serverless deploy. It fails open by design: an outage degrades to unlimited, never to downtime.

Curator notes as first-class retrieval content

My own commentary is chunked and embedded alongside article text, and boosted at retrieval time. The archive answers with what I thought, not just what I read — that judgment layer is the product.

Want the long version? The case study covers the architecture decisions, the eval methodology, and what I'd change.

David's Archive

One person's reading, compressed into a queryable knowledge base. Built with retrieval, evals, and a learning loop — and instrumented end to end.

Explore

  • Library
  • Stats
  • Evals
  • About

Project

  • RSS feed
  • Source on GitHub
  • Case study

Weekly digest

New posts, once a week. No tracking pixels.

© 2026 David's Archive · answers cite their sources