Tree-sitter

Tree-sitter is a parser generator built to be fast enough and forgiving enough to parse every file in a real codebase, again and again as you type, including the many files that do not currently compile. It produces a syntax tree the same way any parser does, but it is designed for the messy, moving conditions of a live project rather than the clean conditions of a finished build.

The problem editors had

To see why tree-sitter exists, start with the older way parsers were used. A traditional parser was written to serve a compiler. A compiler runs on demand, on code you have declared finished, and it parses each file once, from the first character to the last, to build the program. That fits a compiler perfectly. It fits a code editor very badly, and the mismatch is the whole story.

An editor wants to understand your code continuously, while you are writing it, to color the keywords, mark the errors, jump to a definition, and fold a function. That means it cannot parse once and be done. It has to keep a fresh understanding of the file after every keystroke. And crucially, the file it is parsing is almost never a valid program. Halfway through typing a function you have an open brace with no close, a half-written name, a line that stops in the middle. A compiler-style parser meets the first of these and stops with an error, which is exactly the wrong behavior for a tool that has to keep working while the code is broken. Editors needed a parser that is quick enough to re-run constantly and steady enough to make sense of incomplete code, and that is the gap tree-sitter was built to fill.

Incremental parsing

The first thing tree-sitter does differently is incremental parsing. A plain parser, asked to re-parse after you type one character, would read the entire file again from the top, throwing away everything it knew and rebuilding the whole tree. For a small file that is fine. For a large one, done on every keystroke, it is wasteful, and the waste shows up as lag.

Incremental parsing avoids the waste by reusing what has not changed. Tree-sitter keeps the syntax tree from the last time it parsed. When you make an edit, it is told which small span of text changed. It then updates only the part of the tree affected by that span and keeps the rest of the tree exactly as it was. Type a character inside one function and only that function’s corner of the tree is redone; the hundred other functions in the file are left untouched.

Think of it like fixing one typo in a printed book. You do not reset every page. You reprint the one page that changed and leave the rest on the shelf. Because the work is proportional to the size of the edit rather than the size of the file, tree-sitter can keep up with typing even in enormous files, which is exactly what an editor needs. The tree is always current, and keeping it current is cheap.

The same trait pays off outside the editor, in any tool that keeps a standing understanding of a codebase. If a tool has already parsed a project and a developer changes a handful of files, the tool does not have to re-read the whole project to catch up. It can re-parse just the files that changed, and within each file only the parts that changed, then update its understanding from there. Incremental parsing turns keeping up to date from a heavy, whole-project job into a light, edit-sized one, which is what makes it realistic to track a repository as it moves rather than taking a stale snapshot now and then.

Error recovery: parsing broken code

The second and more striking thing tree-sitter does is error recovery. A plain parser treats a syntax error as a wall. It reaches text that fits no rule, reports the error, and stops, leaving you with no usable tree past that point. That is defensible for a compiler, which will not build a broken program anyway. It is useless for a tool that must understand code while it is being written, because such code is broken most of the time.

Tree-sitter refuses to give up. When it meets something that does not fit the grammar, it does not throw the whole tree away. It marks the troubled spot as an error node, makes its best guess about where the valid structure resumes, and keeps parsing the rest of the file. The result is a tree that is mostly correct with a small damaged region, rather than no tree at all. Picture a file where one function is half-written:

function ok() { return 1 }

function broken( {

function alsoOk() { return 2 }

The middle function is malformed, missing its parameter list and its body. A plain parser might stop there and never see the third function at all. Tree-sitter parses the first function correctly, marks the middle one as containing an error, recovers, and parses the third function correctly too. You still get a tree for the two good functions, which is exactly what a tool needs to keep offering help around the broken part instead of going dark the moment the code stops compiling.

Tree-sitter treats a syntax error as a small damaged region to work around, not a wall to stop at. That single choice is what lets it understand code that does not yet compile.

One engine, many languages

The third thing that makes tree-sitter widely useful is that it is language-agnostic. Tree-sitter itself does not know any particular programming language. It is the engine that generates and runs parsers. The knowledge of a specific language lives in a separate grammar, written in tree-sitter’s notation, that describes that language’s rules. There is a grammar for one language, a different grammar for another, and the same tree-sitter engine drives them all.

This separation is what makes the approach scale across a whole ecosystem. Because a grammar is a self-contained description rather than a bespoke parser welded to one language, grammars can be written and shared by the community, and a tool that speaks tree-sitter gains a new language simply by loading that language’s grammar. You do not build a fresh parser per language from scratch, which is the trap that keeps most tools tied to one or two languages. You build once against tree-sitter and add languages by adding grammars.

The payoff is uniformity. Every language, once it has a grammar, produces the same kind of syntax tree through the same engine, with the same fast incremental updates and the same error recovery. A tool can then walk a JavaScript tree and a Go tree and a Python tree with the same machinery, because they are all trees of nodes and children in the end, differing only in the grammar that produced them.

There is a matching benefit for asking questions of these trees. Tree-sitter comes with a small way to describe patterns of nodes you want to find, so that instead of writing traversal code by hand for every language you can state, once, the shape you are looking for, such as a function definition or a call, and let the engine find every match. Because the trees share a common form, the same kinds of pattern carry across languages with only the node names changed to suit each grammar. That keeps a multi-language tool from turning into a dozen unrelated tools wearing a trench coat. It stays one system that speaks many languages, rather than many systems bolted together.

Write your tool against tree-sitter once, and each new language is a grammar you load, not a parser you build. That is what makes covering a dozen languages tractable at all.

Why this is what you want for a whole repository

Now put these three properties against the real job of indexing an entire codebase, and see how neatly they fit. A real repository is large, it is written in several languages, and at any given moment some of its files are in a broken or half-finished state, especially the files a developer is actively editing. A tool that wants to build an understanding of the whole thing has to cope with all three conditions at once.

Tree-sitter’s three answers line up exactly with the three conditions. Speed and incremental parsing handle size, so parsing thousands of files, and re-parsing them as they change, stays practical rather than glacial. Error recovery handles brokenness, so a file that does not compile still yields a usable tree and still contributes what facts it can, instead of being skipped. Being language-agnostic handles the mix of languages, so one indexing engine can cover a polyglot repository without a separate parser project for each language. A parser built only for finished, single-language, compiling code would fail the real repository on all three counts. Tree-sitter was shaped, from the start, for precisely these conditions.

The result is that tree-sitter can turn every file in a project, valid or not, in whatever mix of languages, into the nested tree of nodes that everything else is built on. It is the practical bridge between the idea of a syntax tree and the reality of a working codebase full of imperfect, changing files.

It is fair to ask what tree-sitter deliberately does not do, because its focus is part of why it is dependable. It parses. It gives you an accurate tree of the structure in a file, quickly, even when the file is broken, in many languages. It does not try to understand meaning beyond the grammar. It does not decide which definition a name refers to when several exist, it does not follow a call from one file into the file that defines the function, and it does not know anything about the rest of the project while it looks at any one file. Those are real and important jobs, but they are separate jobs, layered on top of the trees rather than done by the parser. Tree-sitter draws the reliable outline of each file and stops there, on purpose, and that clean boundary is exactly what makes it a solid foundation to build the harder analysis on.

This division of labor is why tree-sitter has spread so widely. It was created for a code editor, where its speed and its tolerance for broken input mattered most, and it has since been adopted far beyond that first home, by editors, documentation tools, static analyzers, and code search systems, wherever a program needs an honest, fast, forgiving tree for source in many languages. It earns its place by doing one thing well and leaving room above it for everything else.

Where Koragraph uses it

This is the foundation Koragraph MCP is built on. Koragraph MCP is a local server that gives AI coding agents a persistent code knowledge graph of your repositories, and it parses those repositories with tree-sitter across twelve languages: C, C++, C#, Go, Java, JavaScript, PHP, Python, Ruby, Rust, Swift, and TypeScript with TSX. It runs entirely on your machine, with no cloud dependency, so no source code leaves the computer while it parses. Using tree-sitter is what lets a single tool read a whole polyglot repository, including files that are mid-edit and do not currently compile, and still produce a consistent tree for each one.

Everything you have read so far explains only the first move: turning flat text into trees, reliably, at the scale of a real project. A tree, though, is still just the structure of one file. The trees are the raw material, not the finished map. The next step is to walk those trees and pull out the concrete facts they hold, starting with the plainest one: what each file declares, meaning the functions, classes, and other named things it defines. That step is called declaration extraction, and it is the next idea in this hub, on the way to the ingest pipeline that assembles all of these facts into a single graph.