I Read Karpathy's Tweet, Found a YouTube Video, and Had a Working 'Second Brain' by Sunday
A Persistent Wiki: The Difference Between Retrieving Knowledge and Compiling It
Andrej Karpathy’s idea for building a personal knowledge base with LLMs is short. He describes the concept, architecture, operations, and even some optional CLI tools. What he deliberately leaves out is which tools to use.
In the post, he’s explicit about it: the document’s job is to communicate the pattern, and your LLM agent figures out the rest. Different people, different stacks, different needs. He wanted everyone to build their own version, not copy his.
When the tweet went viral, I read the idea file twice and understood the concept completely. I had no idea how best to build it. Then last weekend, I found Corey Ganim and Nick Spisak’s walkthrough on YouTube. By Sunday, I had a working system. This article is about both dots and the line between them.
What Karpathy’s concept is actually saying
The problem he’s solving is subtle enough that it’s worth stating plainly. Most people’s experience with LLMs and documents is retrieval: you upload files, the model finds relevant chunks at query time, and generates an answer. Nothing accumulates. Ask a question that requires synthesizing five sources, and the system assembles an answer from scratch, forgets it did so, and will do it again next time. The model has no memory of having thought this through before.
His alternative: use an LLM to compile a persistent wiki, not just index raw documents. The LLM reads your sources, extracts what matters, and builds a collection of interlinked markdown files that it maintains as you add new material. When a new source arrives, the LLM doesn’t just file it. It updates existing pages, notes where the new data contradicts old claims, strengthens cross-references, and logs what changed. The synthesis is pre-built. You’re querying something that has been actively constructed, not something that retrieves raw text and hopes for the best.
Three layers:
raw (your source documents, untouched, your ground truth)
wiki (LLM-maintained markdown files, entirely the LLM’s domain)
outputs (queries, analyses, anything worth keeping from a session)
A schema file tells the LLM how the wiki is structured and what to do on every ingest, query, and health check. That schema is the key configuration: it’s what turns a general-purpose AI agent into a disciplined wiki maintainer.
The idea is clean. The question is what you actually build it with.
How Corey and Nick closed the gap
After the tweet went viral, Corey Ganim and Nick Spisak put out a walkthrough that translates Karpathy’s abstract pattern into a concrete, end-to-end build using Claude Code and Obsidian. I followed it step by step. (Link to their video at the bottom of this post. Absolutely worth 20 minutes of your time.)
The specific tool choices matter because they fit the pattern so naturally. Obsidian is a markdown-based note-taking tool with a graph view that renders the connections between pages as a visual network. That graph view turns out to be the best way to watch the system actually working: you can see the wiki building in real time as the LLM processes your sources. Claude Code handles the agent operations, and a pre-built skill that Corey and Nick made available handles the scaffolding automatically.
The setup sequence, following their walkthrough: install Obsidian, install the Obsidian Web Clipper Chrome extension (which converts any web article to markdown in one click and drops it into your vault), fire up Claude Code, and run the second brain skill. A guided wizard asks you to name the vault, choose where to store it, and configure any optional tools. It creates the full directory structure (raw, wiki, outputs), generates the CLAUDE.md configuration file, and gets you into a working state in under five minutes.
After that, the daily workflow is three commands. An example of a typical personal workflow:
Ingest: clip articles into raw using the browser extension, optionally pull YouTube transcripts using a CLI tool called summarize, then run the ingest command. Claude Code reads everything in raw, writes summary pages to the wiki, updates the index, and builds the connections.
Query: ask the system a question, get a synthesized answer drawn from the wiki, file anything useful back in as a new page.
Lint: run a health check that finds contradictions between pages, flags gaps, and suggests what to read next.
The moment the abstraction becomes real
Before you run your first ingest, Obsidian’s graph view shows a handful of orphaned nodes: the scaffold files exist, but nothing is connecting them. After you ingest even a single source, that changes. The LLM reads the article, writes a summary page, creates entity pages for the people and concepts mentioned, and cross-references them. The graph fills in. You can follow the links.
I watched it happen with one Wikipedia article and immediately understood what Karpathy meant when he described the wiki as a compiling artifact. The connections weren’t something I made. The system made them, and they were already there when I went to look.
What compounding actually looks like
Day one, the wiki is as useful as a well-organized bookmark folder. Day 30, it’s better than your memory on the topics you’ve been feeding it. The compounding happens in two directions: sources accumulate as you ingest them, and your own queries accumulate too. When you ask a question and get a useful analysis, you file that output back as a new wiki page. Your explorations don’t disappear into chat history. They stay in the knowledge base and make the next query more capable.
At around 100 sources, Karpathy notes from his own use that the LLM handles complex synthesis questions well without any embedding infrastructure. The index file is enough for navigation. No vector database, no embedding pipeline, just markdown and an agent that knows how to read an index and drill in. You can build this without any backend infrastructure at all.
The thing Karpathy understood that’s easy to miss
He released a deliberately incomplete idea because he knew the right implementation would look different for everyone. Some people will use Obsidian and Claude Code. Others will use a different editor, a different agent, a different folder structure. The pattern is what transfers. The specific tools are secondary.
That said, if you want a concrete starting point, Corey Ganim and Nick Spisak’s walkthrough is the clearest implementation I’ve seen (I did check many others). Follow it, get the system running, and then adapt it to how you actually work.
Pick a domain, clip three articles, and run your first ingest. The graph starts filling in from there. Or convert your project notes to markdown files, dump them in the ‘raw’ folder, and run your first ingest.
I already picked up a few fresh insights from my files after running the synthesis command. Off to a great start. You should give it a go.
Happy weekend!
~~~