Can Graph-Based Context Engineering Substitute for Model Scale?

1. Hypothesis

The claim under test is narrow and falsifiable.

On bounded repository-level code tasks, a cheap model given structured context derived from a code knowledge graph solves as many tasks as a frontier agentic coding system given ordinary grep search over the same repository.

Each part of that sentence is load-bearing. The outcome is tasks solved, not output quality, latency, or developer preference. The scope is bounded tasks, meaning ones with a statement, a pinned commit and a mechanical oracle, and not open-ended engineering. And the subject is a deployed configuration rather than a set of model weights: the cheap condition includes its retrieval layer, and the frontier condition includes its agentic search loop.

The reason to ask is economic. If the hypothesis holds, the cost of the retrieval layer is paid once and amortised, while the cost of the frontier tier is paid per task forever. If it fails, it should fail visibly, and the failure should say which tasks resist cheap models.

A second question falls out of the same design, and it turns out to carry the result: does structured context help both tiers equally? A retrieval layer that lifts every model is a convenience. One that lifts a cheap model to the level of an expensive one while doing nothing for the expensive one is a substitution, and substitution is the economically interesting claim.

2. The system under test

The graph condition supplies the model with context assembled by Koragraph, a persistent and queryable representation of the repository, rather than from a text search over it. The representation records the entities a codebase contains and the relationships between them. It is built once per repository revision and reused across every task run against that revision. Retrieval against it combines semantic similarity with traversal of those relationships, so the context assembled for a task can include code related to the task that shares little vocabulary with its description. That is the property grep cannot offer, and the one this experiment prices.

We describe it at this level on purpose. What the experiment needs in order to be reproducible is the interface: each condition receives a context block, the blocks are held to a shared byte budget, and what differs between conditions is how the block was chosen. Nothing in the analysis depends on the internal construction of the representation, and a reader who substitutes a different graph retriever behind the same interface is running the same experiment.

The grep condition is the comparator a working engineer would actually reach for: search the repository for terms drawn from the task statement, rank the hits, and pass the top-ranked file contents forward under the same byte budget.

3. Experimental design

3.1 The four conditions

The design is a 2×2 factorial. One factor is model tier: a frontier agentic coding system with an unrestricted tool loop, against a cheap model invoked in a single-shot loop with bounded retries. The other factor is context source, grep against graph retrieval.

The two models stand in for the ends of a deployment decision rather than for their tiers as a whole. The frontier condition is claude-opus-5 driven through Claude Code, which is what supplies the unrestricted tool loop: within a single cell the system can search the repository, read files, execute the repository’s own tests, and revise before it emits a patch. The cheap condition is deepseek/deepseek-v4-flash-0731, called through an inference gateway in a single-shot loop with bounded retries and no tool access, so everything it learns about the repository arrives in the context block it is handed. That difference in loop architecture is deliberate, because it is the difference a buyer is actually choosing between, and it is also why the diagonal is confounded.

Both run at temperature 0, and costs are computed against rate table version 2026-08-07.1. One model per tier is a real limit on what “tier” can mean here: any tier effect reported below is confounded with the identity of the model standing in for it.

The primary endpoint is the diagonal, B against A. It is the comparison the hypothesis is about, and it is also the most confounded comparison in the matrix, since model tier, context source and loop architecture all move at once. We treat it as quotable only because the off-diagonal cells are run as well, so the decomposition that would explain any difference is published beside it. The within-tier contrasts, B against D and C against A, vary retrieval alone and carry no such confound.

3.2 What each condition gets, and where the remaining asymmetry lies

The most common objection to a benchmark published by the builder of one of its conditions is that the comparator was hobbled. Stated so it can be checked:

• One context budget, shared by every condition that receives a block. The budget is 100,030 bytes, realised within 0.9% across the cells for which per-cell records exist. Whatever a structured block spends, the file block cannot. Condition A is the exception and is not bound by it: it searches agentically and is handed no context block at all.
• The same retry allowance. Three attempts per cell for both tiers.
• The same feedback between attempts. Both tiers see a compile gate and an executed run of the repository’s own suite. The feedback arrives as an executed command result and never as a proof command in the prompt, so no condition is told where the task’s code lives.
• The same oracles, hidden from both. Neither the fail-to-pass nor the pass-to-pass bundle is visible to any condition.

One asymmetry survives, and it favours the frontier tier. The cheap conditions run under a 131,072-token output ceiling, while the frontier conditions carry no harness ceiling at all. That figure is the result of a repair made during validation: the ceiling was originally format-dependent and in one format sat at 4,096 tokens, close enough to a correct answer to matter. We raised the cheap ceiling rather than lower the frontier one, on the view that a finite ceiling imposed on one condition alone is a handicap and the honest repair is to remove it where we can.

3.3 Draws, and what counts as solving a task

Every condition receives three independent draws on every task. A task counts as solved by a condition when a majority of its three draws pass the oracle. Three draws on all four conditions departs from the pre-registration, which specified a single draw on the frontier conditions. Equalising the draw count removes an asymmetry between tiers and it also lowers the frontier conditions’ apparent per-draw success rate relative to a single-draw measurement, because three draws expose variance that one draw hides. We prefer the fairer number and record the deviation here rather than in a footnote.

The full matrix is 11 tasks × 4 conditions × 3 draws, or 132 cells. It was executed three times, on separate machines with separate credentials, and returned the same tasks-solved outcome on each execution.

3.4 Grading, and the difference between a failure and a void

Each task carries two executable oracles: a fail-to-pass check that the intended behaviour now holds, and a pass-to-pass check that previously-passing behaviour still does. A patch is graded against a fresh checkout at the pinned commit, in a fixed order. A candidate diff that touches the fail-to-pass paths is a protocol violation and scores zero.

A scored class is evidence about the model and counts against its condition. A void class means the instrument failed to deliver the treatment, which makes it missing data rather than a result. Voids are excluded from every rate and retried under a fixed allowance. Scored cells are final and are never re-run, because re-running scored cells until they pass is how a harness manufactures a result. The rule this enforces, stated so it can be held against us: a task may be rejected only for a cited mechanical defect, never because a model failed it. In the reported matrix, two arm-D cells voided and both scored on retry, so no condition carries a surviving void and no contrast is withheld.

3.5 Execution order

Cells are executed in a seeded shuffle rather than in task-major order. A fixed arm-minor order gives every condition the same position in every task’s sequence, so anything position-correlated lands on the same condition every time and becomes indistinguishable from a treatment effect. A cold build cache absorbed by whichever condition runs first would do it; so would a rate limit that bites late. The shuffle seed is recorded, so order is random with respect to condition and still exactly reproducible.

4. The task suite

Eleven tasks are drawn from two open-source repositories at pinned commits: a Java web application (Spring PetClinic, Apache 2.0, at f182358) and a Python forum library (django-machina, BSD 3-Clause, at 5ecfad3). Both are real projects with real test suites, and every task is derived from a defect or a missing capability genuinely present at the pinned commit rather than injected for the benchmark.

Statements are written from observable behaviour, meaning what a user sees go wrong, without reference to the patch that fixes it. Mechanical checks reject any statement naming a target file or type. This matters for a retrieval experiment: a statement that names its own gold file hands the answer to every condition and leaves the retrieval treatment nothing to measure.

The suite spans boundary faults, configuration, localisation across twenty-six translation catalogues, shared contracts, ordering, input validation, a permission leak and two error contracts. Two of them — M3J and M3P — are negative controls: single-file tasks with no ripple beyond the file that carries the defect. They are included because a retrieval treatment should show little advantage where there is nothing to traverse, and a corpus that reports an advantage everywhere is more likely measuring something other than retrieval.

4.1 What the construction protocol caught

A benchmark’s credibility rests on the gates it survived, and gates that are declared rather than executed are worth nothing. Each task passed nine gates in sequence. Every one of them found something, and several found defects in work an earlier gate had blessed.

Twelve tasks were authored and one was withdrawn for a cited mechanical reason: its patches rewrite an error code the repository’s own suite asserts on, so a behaviourally correct candidate records a regression in every condition. We report the withdrawal rather than only the surviving count.

5. Results

5.1 The interaction

The clearest result in the matrix is not the diagonal but the difference between the two within-tier contrasts, and it is clean in a way the diagonal is not. Within a tier, the model, the loop and the draw count are held fixed, and only the context source changes.

Within the cheap tier, replacing grep with graph retrieval takes the model from 2/11 tasks to 9/11 — seven tasks won, none lost. Within the frontier tier the same substitution produces 9/11 against 9/11, one task traded in each direction.

The two negative controls qualify this. On M3J and M3P, single-file tasks with no ripple, the cheap condition still solves 3/3 with graph context against 0/3 and 1/3 with grep. If retrieval only helped where traversal helps, these tasks should have shown little difference. They did not, so the effect is not attributable to traversal alone, and part of the cheap tier’s advantage plausibly comes from being handed the right file at all.

5.2 The diagonal

On the primary endpoint, B and A each solve 9 of 11 tasks, at $0.02662 and $2.81936 per solved task, a ratio of 105.9×. Across all eleven tasks, seven are solved by both, two by A alone (M1P, M2P), two by B alone (J1, M2J), and none by neither.

The two conditions therefore fail differently rather than one dominating. A loses J1, a single-file boundary fault, and M2J, a contract change with three dependent files. B loses M1P, whose correct patch spans twenty-six translation catalogues, and M2P, a permission-visibility change across a shared type. M1P looks like a capability limit rather than a retrieval limit: the cheap condition locates the relevant code and then fails to express an edit of that breadth.

We state plainly what this does and does not support. Two discordant pairs in each direction is a balanced result, and a paired comparison at this sample size cannot resolve a difference of practical size in either direction. It establishes that no difference was detected, not that the conditions are equivalent, and by the same token it does not place the frontier condition ahead. We decline to round a null result to parity. The defensible sentence is that the two conditions solved the same number of tasks on this corpus, at costs differing by two orders of magnitude, and that the cheap condition’s failures are concentrated in tasks requiring wide, uniform edits.

5.3 How each condition fails

Outcome classes separate the two cheap conditions further. Condition D’s dominant scored failure is a pass-to-pass regression, 13 of its 28 cells, against 7 of 29 for condition B on the same model and the same loop. A cheap model given poor context does not only miss the fix; it edits code it should not have touched and breaks behaviour the task did not ask it to change. The frontier conditions record one such regression apiece. This is the failure mode a reader deploying a cheap tier should price.

5.4 Cost, and why the ratio is a floor

Frontier costs are modelled from token counts at published list rates; cheap-tier costs are metered. The distinction is material and we do not blur it. The frontier system runs against a subscription that emits no per-call price, so a metered figure for it does not exist. Indexing is a separate line item and is not folded into any per-task cost.

The direction of the residual error is measurable, and it runs against our own headline. Across the 22 frontier cells carrying per-cell records, the executing engine’s own reported cost exceeds the modelled figure by a median of 1.22× for condition A and 1.26× for condition C. The modelled basis therefore understates frontier spend by roughly a fifth to a quarter, and 105.9× is a lower bound on the cost ratio. We report the modelled figure anyway, because it is the one a reader can derive from a published rate table.

The cost comparison is an accounting consequence of that rate table and the two loop architectures. It is not a hypothesis this experiment could falsify. The quantities a reader can use are the ratio per solved task and the observation that the frontier conditions spend their advantage on tokens consumed by search the graph condition does not need to perform.

5.5 An exploratory note on retrieval quality

This subsection is exploratory and should be read that way. It was not part of the pre-registered design, the study is not powered to settle it, and nothing in the results above depends on it. Cells carry a record of whether the context block the model received contained the file or files the reference patch modifies.

The two right-hand columns point in opposite directions between tiers. Where the retrieval missed, the cheap condition solved none of those cells and the frontier condition solved all three such cells of its own. If that pattern held at scale it would suggest that what model scale buys on these tasks is partly the ability to recover from being handed the wrong context, paid for in tokens rather than in tasks. The design cannot settle any of that. Three cells cannot carry a claim, and recall is in any case not sufficient on its own: J5 had a gold file served in 3/3 cells and was still lost 0/3 by the cheap condition.

6. Threats to validity

7. Related work

Benchmarks for repository-level code work are easy to build wrong, and the errors are systematically flattering. An audit of SWE-bench found that 32.67% of successful patches had their solution present in the issue text and 31.08% passed tests too weak to distinguish a correct patch from an incorrect one; removing those instances dropped one agent’s resolution rate from 12.47% to 3.97%. Leaked solutions and permissive oracles are the two failure modes our blind statements with mechanical leak checks, and our adversarial degenerate rounds, are built to target.

Graph-structured retrieval for code has been explored as a way to supply models with dependency context rather than whatever a text search happens to return. A more recent line questions how much of that benefit survives against well-tuned grep baselines, finding that grep generally outperforms vector retrieval and that the result depends heavily on the surrounding agent harness. Our result is consistent with both positions once the tier interaction is taken into account, and it suggests why the harness dependence they report should be expected. An agentic harness can re-query, so it converts a retrieval miss into token spend, while a single-shot cheap model converts the same miss into a failed task. A grep baseline is competitive for a sufficiently capable agentic searcher and is not competitive for a cheap single-shot one, so any evaluation that fixes one tier will report one half of that.

Two findings bear directly on ours. Work on retrieval utilisation across model scale reports that models of 7B parameters and below fail to extract the correct answer from oracle retrieval on the large majority of questions they cannot already answer, which is a utilisation bottleneck rather than a delivery one. Our cheap condition is far larger than 7B and clears the utilisation bar on nine of eleven tasks, and its two losses look like utilisation failures, which is what a utilisation ceiling predicts. Separately, work on a structural codebase index inside a coding agent reports that the index lands at a lower cost per solved task than agentic grep. We reach a compatible cost conclusion by a different route, and would qualify the framing, since on our corpus the structural advantage appears even on single-file negative controls.

8. Artifact availability

The task suite is released as Koragraph_test_suite_1. For each of the eleven tasks it contains the statement as given to every condition, the pinned commit, both oracle commands, the fail-to-pass test files with recorded checksums, two independently authored reference solutions, and the executable degenerate patches an oracle must reject. The grading protocol and the failure-versus-void taxonomy are included, so a task can be graded exactly as it was graded here. Both upstream repositories carry permissive licences and the released patches are redistributed under them with attribution. Neither upstream project endorses this suite.

What is released is the corpus and its oracles, not the retrieval implementation. The reproducible unit of this experiment is the interface, meaning a task, a pinned repository, a byte budget and a context block, and any retriever placed behind that interface is running the same experiment. A reader who disputes the result has what is needed to re-measure it against a retriever of their choosing.

Eleven tasks is a small corpus, and we prefer a small one that carries executable oracles, adversarial degenerate patches and a stated construction protocol to a larger one that carries none of them. The suite is built to grow: contributions clearing the same nine gates are welcome, and independent replication is the reason it is public.

9. Conclusion

Structured context substitutes for model scale in one tier and not the other. Given graph-derived context, a cheap model went from 2/11 to 9/11 tasks on this corpus, winning seven tasks and losing none, while the same substitution moved a frontier agentic system not at all. On the pre-registered diagonal the cheap-plus-graph condition matched the frontier-plus-grep condition at 9/11 tasks each, for one hundredth of the cost per solved task, while failing on a different pair of tasks.

The result we would defend is the interaction, because it is the one measured without confounds. The diagonal we report with its cost, its decomposition and its statistical limits attached, and not as a claim of equivalence. Whether the substitution holds at a larger corpus, on larger repositories, and across more languages is the obvious next question, and this design — a factorial with the off-diagonal cells actually run and a published corpus any retriever can be measured against — is what makes that question answerable rather than rhetorical.

References

1. [1]Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, Song Wang. SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv:2410.06992, 2024.
2. [2]Anthropic. Claude Code. claude.com/claude-code, 2026. Agentic coding CLI; the harness for the frontier conditions.
3. [3]Ishaan Bhola, Adithyan Krishnan, Sravanth Kurmala, Mukunda NS. Code Isn’t Memory: A Structural Codebase Index Inside a Coding Agent. arXiv:2606.22417, 2026.
4. [4]DeepSeek. DeepSeek V4 Flash (deepseek/deepseek-v4-flash-0731), 2026. Served through the OpenRouter gateway.
5. [5]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. arXiv:2310.06770.
6. [6]Akhil Katakam. Koragraph_test_suite_1: eleven repository-level code tasks with executable oracles, 2026.
7. [7]Morgan Aubert and contributors. django-machina: a Django forum engine. BSD-3-Clause. Pinned at commit 5ecfad3.
8. [8]Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, Dong Yu. RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph. ICLR 2025. arXiv:2410.14684.
9. [9]Sanchit Pandey et al. Can Small Language Models Use What They Retrieve? An Empirical Study of Retrieval Utilization Across Model Scale. arXiv:2603.11513, 2026.
10. [10]Sahil Sen, Akhil Kasturi, Elias Lumer, Anmol Gulati, Vamse Kumar Subbiah. Is Grep All You Need? How Agent Harnesses Reshape Agentic Search. arXiv:2605.15184, 2026.
11. [11]Spring Projects. Spring PetClinic. Apache-2.0. Pinned at commit f182358.
12. [12]Hongyuan Tao et al. Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks. NeurIPS 2025. arXiv:2505.16901.
13. [13]Martin Vogel, Falk Meyer-Eschenbach, Severin Kohler, Elias Grünewald, Felix Balzer. Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP. arXiv:2603.27277, 2026.
14. [14]Baoyi Wang et al. Better Call Grep: Evaluating and Improving Grep-Like Lexical Retrieval for Repository-Level Code Completion. ISSTA 2026. arXiv:2601.23254.