The ingest pipeline

Ingest is the single pass that turns a pile of source code into a graph. It walks your repositories, parses each file into a structured form, pulls out the declarations, works out every connection between them, reads the git history for how they change together, and writes all of it into the local store. Each stage feeds the next, which is why it is a pipeline rather than a bag of separate tools.

What a pipeline is

A pipeline is a sequence of stages where the output of one stage becomes the input of the next. The name comes from plumbing, and the picture is exactly that: material flows in one end, passes through a fixed series of steps in order, and comes out the other end transformed. A factory line is the same idea. A car body arrives, one station adds the doors, the next adds the wheels, the next paints it, and each station does one job and hands the result along. No station tries to build the whole car.

The value of arranging work this way is that each stage can be simple, because it only has to do one thing and it can trust that the stage before it has already done its part. The parser does not have to worry about git history. The step that reads git history does not have to understand how to parse a file. Each does its own narrow job on the output it is handed. When you read that a tool ingests your code, this ordered flow is what the word is naming: raw files in one end, a finished graph in the store at the other, through a set line of stages.

The stages, in order

Here is the whole pipeline laid out as the ordered walk it is. Each step assumes the one above it has finished, and produces exactly what the one below it needs.

1. Discover the files. Before anything can be parsed, the tool has to find what to parse. It walks the folders of every repository you have pointed it at, gathers the source files, and skips what is not source: build output, dependency folders, binary assets, anything that is not code you wrote. What comes out of this stage is a plain list of files worth reading.
2. Parse each file into a tree. A source file is, to a machine, just a long run of characters. Parsing is the act of reading those characters according to the rules of the language and producing a structured tree that says what the code actually is: this is a function, this is its name, this is its body, inside the body is a call. That tree is called a syntax tree, and it is the form every later stage works from. Koragraph does this parsing with tree-sitter, a parsing engine that reads real source code and builds such a tree, and it does so across twelve languages: C, C plus plus, C sharp, Go, Java, JavaScript, PHP, Python, Ruby, Rust, Swift, and TypeScript together with its TSX variant.
3. Extract the declarations. With a tree in hand, the tool reads off the things the file declares: every function, every class, every method, every type. Each becomes a node, recorded with what kind of thing it is, what it is called, and where it lives. These declarations are the nodes of the graph. Nothing can be connected yet, because you cannot draw an edge until both of its endpoints exist as things on record.
4. Resolve the edges. Now that the declarations exist as nodes, the tool works out the connections between them. When one function calls another, that is an edge. When one file imports another, that is an edge. When one class inherits from another, that is an edge. Resolving means matching each reference to the actual declaration it points at, so a call to a name becomes a real link to the one function that name means, not just a note that some name appeared.
5. Resolve the connections across services. Real systems are not one program. They are several programs that talk to each other, and the links between them do not look like plain function calls. One service sends an HTTP request to another. One calls another over gRPC. One publishes a message to a topic that another listens on. A Docker Compose file wires the services together. Koragraph resolves these cross-service edges too, including HTTP, gRPC, package imports, message topics, Docker Compose, framework routes, and even Python runtime tracing, so the graph does not stop at the edge of any one repository.
6. Mine the git history. Beyond what the code says right now, there is what its past reveals. The history of changes records which files were edited together, over and over, across the life of the project. Two files that keep changing in the same commit are connected in a way no line of code states outright. Koragraph mines the git history to find this co-change relationship and records it as its own kind of edge.
7. Write it all to the store. The finished nodes and edges are written into the local SQLite database, the single file where the graph rests between sessions. Once this stage completes, the graph is on disk and ready to be asked questions, and it will still be there the next time the tool runs.

Why the order is not negotiable

It is worth dwelling on why these stages must run in this sequence, because the order is not a matter of taste. Each stage genuinely depends on the finished output of the one before it, and running them out of order would not merely be slower, it would produce a wrong or empty graph.

You cannot parse a file you have not found, so discovery comes first. You cannot extract declarations without a syntax tree, so parsing comes before extraction. This next dependency is the one people most often miss: you cannot resolve an edge until both of its endpoints exist as nodes. An edge is a link from one thing to another thing, and a link needs two ends. If the tool tried to resolve a call before it had extracted the function being called, there would be nothing on record for the call to point at. So every declaration in scope has to be found and recorded before any edge is drawn. That is why extraction, across all files, comes before resolution.

Cross-service resolution sits after ordinary resolution because it is the harder, broader version of the same job and it leans on the same nodes being in place first. History mining can run once the files are known, since it reads the record of commits rather than the syntax trees, but its results are only meaningful once there are nodes to attach them to. And writing to the store comes last for the plain reason that you write down a result after you have it, not before. Read as a whole, the order is just cause before effect, all the way down.

An edge needs two endpoints, so every declaration has to become a node before a single connection can be resolved. That one rule fixes the shape of the entire pipeline.

The first run and what it costs

The first time you ingest a repository, the pipeline does the full amount of work, because nothing is on record yet. Every file is discovered, every file is parsed, every declaration is extracted, every edge is resolved, and the whole history is mined. With Koragraph the command that starts this is koragraph ingest /path/to/repo, and after it finishes there is a companion check, koragraph doctor, that confirms the graph came out in good health. From that point on the graph exists in the store, and the question becomes how it keeps up with code that will not stop changing.

Keeping the graph current

A codebase is a moving thing. People add functions, delete files, rename classes, and rewire the connections between them every day. A graph built once and never updated would slowly drift away from the truth until its answers could not be trusted. So the graph has to be kept current, and the sane way to do that is not to rebuild the whole thing from scratch every time one file changes.

The better approach is incremental updating: redo the work only for what actually changed. If a single file was edited, the tool can re-run the pipeline for that file alone. Parse the new version of it, extract its declarations again, and resolve the edges that touch it, while leaving untouched the vast majority of the graph that did not change. This is possible precisely because of the stable identity discussed in the store: each node has a durable handle of its own, so an updated file’s function can be recognized as the same function it was before, its old edges cleaned up, and its new ones drawn, without disturbing anything else.

Incremental updating is what makes the pipeline something you can live with rather than a weekly batch job. The expensive first pass happens once. After that, the cost of staying current is proportional to how much you changed, not to how large the whole codebase is. A one line edit costs about as much as a one line edit should.

The same stable identity is what lets an update survive a rename cleanly. When a file is moved or given a new name, a naive rebuild would treat the code inside it as freshly appeared and forget everything already known about it. Because Koragraph’s store follows file renames, the pipeline can instead recognize that the function in the renamed file is the one it already had on record, keep the edges pointing at it intact, and simply update where it now lives. The practical effect is that reorganizing your folders does not cost you the graph. The tool keeps up with the tidying instead of being reset by it, which is exactly the behavior you want from something meant to run quietly alongside daily work.

Why a pass, and not a live parse

A fair question is why any of this should be a separate ingest step at all. Why not simply parse the code fresh at the exact moment a question is asked, and skip the store entirely? The answer is the difference between preparing and improvising. Resolving the edges of a large, multi-repository system is real work, and the cross-service and history-based connections in particular cannot be seen by glancing at a single file in isolation. They only appear when you look across the whole set of repositories at once. Doing that work up front, once, and saving the result means every later question is answered against a graph that already knows all of these connections, instead of trying to rediscover them under the pressure of a live request.

This is the same reason a library keeps a catalog. It would be absurd to re-sort every shelf each time a reader asks where a book is. You do the organizing once, keep the catalog current as new books arrive, and then any question is a quick lookup. Ingest is the organizing pass. The store is the catalog. The queries an agent makes later are the quick lookups the catalog makes possible.

Ingest pays the cost of understanding the code once and saves the result, so that every question afterward is a lookup against a graph that already knows the answer’s shape.

Where this leads

The pipeline is the spine that the rest of this hub hangs from, because every other idea is really one of its stages seen up close. The parsing stage rests entirely on tree-sitter, the engine that turns characters into a syntax tree. The extraction stage is the art of reading declarations off that tree. The cross-service stage is the study of edges that leap between separate programs. The history stage is the practice of reading co-change out of a project’s commits. And the final stage writes into the SQLite store where the whole graph lives. Any of those is a good next step, and each one is simply this pipeline slowed down and examined at a single station of the line.