Notes on what we are seeing in agent research, from the team building Koragraph.
GPT-6 Astra scored 62.7 percent on ARC-AGI-3 under the benchmark’s standard test harness. Wrapped in OpenAI’s own provider harness, the identical model scored 99.9 percent. ARC Prize published both figures itself this week, describing the result as a score that “increased from 62.7% to 99.9%”, which is the detail worth sitting with. Nobody retrained Astra between those two numbers. Somebody changed what it was allowed to keep.
What actually changed between the two scores
ARC-AGI-3 tests two different things depending on which harness a model runs under. The standard harness hands a model everything it needs and leaves it to decide, on its own, what notes to carry forward as it plays. OpenAI’s provider harness does something different. It preserves the model’s reasoning state between requests and compresses the conversation as it grows, so the model works from a maintained record rather than starting each step nearly cold. ARC Prize frames the two conditions directly: one asks how a model performs on a minimal, provider-neutral interface, the other asks how it performs once it can use the context management its own provider built for it. That second condition is a memory system wearing a benchmark’s clothes, and it is worth 37 points.
The same failure shows up in eighteen other models
Chroma ran a comparable test earlier this year across eighteen frontier models and found every one of them grew less reliable as the input they were given grew longer, even on simple retrieval tasks, and the effect appeared well before any model came close to its stated context limit. The ceiling was never how much a model could technically hold. It was how much of what it held stayed usable by the time an answer was needed. Astra’s harness gap is the same finding, stated as a benchmark score instead of a lab result.
Four fields just found out they were solving the same problem
A paper out of UC Irvine this July, by Ashwin Gerard Colaco and Nada Lahjouji, makes a claim worth taking seriously (arXiv:2607.08032). KV cache eviction, prompt compression, bounding a model’s internal state, and consolidating an agent’s long-term memory look like four separate engineering problems handled by four separate research communities. The paper argues they are one problem: a decision about what to keep and what to drop under a fixed budget, what the literature calls a rate-distortion problem. Its sharper claim is about how all four fail in the same way. Every one of them locks in that decision early, well before anyone knows what the system will actually be asked, and none of them can revisit it later. Sooner or later, each one throws away something a real question turns out to need, whether that is an evicted cache entry, an over-compressed transcript, or a detail folded out of a fixed-size state.
The case for an exact forgetting boundary
A second paper, posted in May and titled “Remember the Decision, Not the Description”, pushes the idea further (arXiv:2605.10870). Most memory systems are built to preserve relevance, or salience, or a faithful summary of what happened. The paper argues that is the wrong target. What actually matters is whether two different histories would ever lead an agent to a different decision. If they would not, they can be collapsed into the same memory, however different they read on the page. The paper calls this an exact forgetting boundary, a calculable line for what a system can safely let go of, rather than a judgement call made on vibes.
Did Astra’s harness solve this, or just outrun it?
OpenAI has not published how its provider harness decides what reasoning state survives compaction as a game gets longer, so this is an open question rather than a settled one. The two papers above give a precise way to ask it, though. If the harness is still choosing what to keep before it knows what the next move will require, it has not solved the problem the July paper describes. It has pushed the point of failure further down the line, into a longer and better-managed context, rather than out of the pattern entirely. The 37 point jump is real and it is impressive. Whether it holds on ARC-AGI-4, or on a task with no clean end state, is a fair thing to want to see before calling the problem solved.
We ran into a version of this while building our own MCP, on a smaller and more concrete surface than a benchmark. A codebase does not sit still, and a memory system that decides what to keep before it knows which function you will touch next runs into the same wall Astra’s harness is quietly managing around. Anchoring memory to the code’s own structure, rather than a compressed transcript, was our way of sidestepping that choice rather than getting better at making it.