54Humans

1. Introduction

Large language model agents now carry out substantial software engineering work, from resolving repository-level issues to multi-hour interactive sessions with a developer. In the interactive setting, a large share of what the developer communicates is not a task but a constraint: a convention the team follows, a failure that must not recur, a step that accompanies a certain kind of change. Such constraints are rarely recoverable from the code itself. Within a session the agent can follow them. Across sessions it cannot, unless they are written somewhere the next session will read.

Existing systems place this responsibility on the agent. MemGPT exposes memory management to the model through function calls, and reflective agents summarise their own experience into a persistent store. Commercial coding agents follow the same pattern: Claude Code maintains a memory directory it can write to and reads instruction files such as CLAUDE.md and AGENTS.md that it can also edit. In each case the model performing the task is also the model deciding what to remember. We observe that this coupling has a cost. An agent engaged in a task records only a fraction of the constraints it is given, and appears to record them selectively, favouring statements that are explicitly framed as rules over those mentioned in passing.

A concrete case from our benchmark illustrates the problem. While asking Claude Code for a small retry helper, a developer adds: “Also, fyi: we’re moving off axios. Don’t use it in anything new, built-in fetch only from now on.” The agent writes the helper correctly. In a later session the same developer asks for a Slack notifier, and the agent’s first line of code is const axios = require('axios');. The agent had saved two other conventions from that project to its memory, and it followed the one that applied. It had not saved this one.

This paper continues an earlier argument, The Authorship Floor. There we proposed that project knowledge is lost at both ends by a single mechanism: recording something is a commitment to keep it true by hand, so developers record only what seems to justify the expense, and what falls below that threshold leaves with the session. We described a memory layer that makes forgetting mechanical by binding each item to the code it is about, argued that this is what makes cheap capture safe, and closed by naming the experiment that paper did not run: tasks selected for friction, a second encounter with the same constraint, and a measure of whether the agent’s work changes. This paper runs that experiment, with the remaining half of the argument, capture without authorship, delegated to a model.

We examine the alternative of separating the two roles. 54Humans is a small language model that runs outside the agent’s control loop and holds exclusive write access to its memory. (The name records the number of attempts it took: the model described here is the fifty-fourth.) After each turn it determines whether the turn contains anything that, if forgotten, would force the user to repeat themselves. On most turns it records nothing. When it does record something, it copies the relevant statement from the user’s message, classifies it, and writes it to a local store that injects it into future sessions. The agent retains read access and loses the ability to write. Our contributions are as follows.

1. An architecture in which a separate local model is the only writer of a coding agent’s persistent memory. It is event driven, adds under 50 ms to the agent’s turns, and uses constant memory in the number of concurrent sessions.
2. 54Humans, a 1.7B-parameter model fine tuned on real and synthetic coding-agent transcripts to abstain on the approximately 90% of turns that contain nothing worth retaining, and to extract rather than paraphrase when it does record.
3. A cross-session benchmark in which constraints are stated once during unrelated work and later sessions are scored by deterministic checks on the code the agent produces.
4. Evidence that 54Humans raises cross-session compliance from 63.0% with the agent’s built-in memory to 94.5%, that most of the gain remains (87.0%) when its entries are delivered through the agent’s own memory, and that across all conditions compliance is determined largely by whether a constraint is delivered to the later session.

2. Problem formulation

A user interacts with an agent over a sequence of sessions on a single repository. Each session begins without the conversational context of earlier sessions. During some session the user states a constraint, typically while requesting unrelated work. In a later session, the agent performs a task whose correct execution depends on it. We say the task exercises the constraint if the agent’s changes touch the behaviour it governs, and that the agent complies if those changes respect it.

A memory writer observes the earlier session and may produce memory entries. A delivery mechanism determines which stored entries appear in the context of the later one. Compliance can fail at three points: the writer may not record the constraint, delivery may not deliver it, and the agent may disregard it once delivered. Our primary metric is the compliance rate, the fraction of exercised (task, constraint) pairs in which the agent complies. We additionally report the delivery rate, the fraction of exercised pairs in which the constraint was present in the later session’s context.

We restrict attention to constraints that cannot be inferred from the repository. An item is worth retaining if and only if forgetting it would force the user to restate it or force the agent to rediscover it at cost. We distinguish five kinds: laws (standing rules), hazards (past failures to avoid), rituals (procedures bound to a trigger), corrections (changes of direction), and open loops (work deliberately deferred).

3. System design

3.1 Overview

The agent runs unmodified except that it has no means of writing to memory. A lightweight hook fires on session events and enqueues the completed turn. A single resident process holds the 54Humans model, drains the queue, and writes accepted entries to the memory store. At the start of each new session the store injects repository-wide entries into the agent’s context. The hook never loads a model, so the user never waits on it, and a single resident process serves every session.

3.2 Triggering

A turn is judged when the user sends the following message. At that point the agent’s reply is complete, and the new message frequently carries information about the previous one, such as a correction or an acknowledgement. An additional trigger at session end handles the final turn. The hook extracts only the visible text of the turn, discarding tool invocations and their output, which in our transcript corpus account for the majority of each turn’s bytes. It then enqueues a job and exits.

A single long-running process owns the model and consumes the queue in priority order, one inference at a time. Because the model is loaded once and shared, memory consumption is independent of the number of active sessions. Jobs are deduplicated by turn and session, and a per-session watermark ensures each turn is judged exactly once.

3.3 Input

Each judgment operates on a bounded packet containing the user’s message and the agent’s reply, the set of files modified during the turn, and the entries already stored for the repository. Modified files are taken from the memory store’s own event log rather than from the agent’s account of its work.

3.4 Judgment

Each judgment is decomposed into three inferences, each given only the information relevant to its question. The first, salience, decides whether the turn contains anything worth retaining, and is given the conversation alone, since whether a statement constitutes a standing constraint is a property of what the user said rather than of the files that changed or the memories already stored. The second, anchoring, runs only when the salience pass produced an entry and the turn modified code. It sees the modified files and may attach an entry to a specific file or declaration, but it cannot add, remove or rephrase entries. The third, consolidation, runs only when memories already exist, sees the full packet, and may update or retire an existing entry. On turns where the salience pass abstains, which is the common case, the judgment is a single inference.

3.5 Deterministic validation

Every judgment passes through a set of checks that involve no model. An anchor must refer to a file that exists or was modified in the turn, or to a declaration named in the packet, and must be named in the entry itself; otherwise the entry is stored at repository scope. At least 30% of an entry’s content words must occur in the session text, a threshold that rejects 0.2% of correct entries in our labelled data while excluding entries whose content originates elsewhere, including restatements of stored memories. Near duplicates and entries with a kind outside the fixed vocabulary are discarded. An entry is attributed to the user only if the user stated it; entries inferred by the agent are attributed to the agent and treated as lower confidence.

3.6 Isolation

54Humans reads the store’s event log in read-only mode, maintains its own state in a separate database, and writes through the store’s standard interface. The component is additive: disabling it leaves all previously written memory readable, and the agent’s behaviour degrades to that of an agent without new memory.

4. Model and training data

4.1 Transcript corpus

We collected 2,132 Claude Code session transcripts from routine development work. Of the messages attributed to the user, approximately 21% were not written by the user but were injected by tooling, including benchmark task descriptions, harness notifications and command wrappers. After removing these, 5,810 user turns remained. Because an entry attributed to the user carries the user’s authority, correct attribution was treated as a first-class requirement of the data pipeline.

The resulting distribution is highly skewed. Approximately 90% of turns contain nothing worth retaining. User messages are bimodal in length, consisting either of short conversational turns or of long turns dominated by pasted logs and plans. Positive turns are frequently marked by the user (“remember that”, “for now”, “going forward”), which makes extraction feasible for a small model.

4.2 Training set and output format

The training set contains 2,835 labelled turns from the corpus and 1,566 synthetic turns designed to cover behaviour that real data exhibits rarely: multiple entries in a single turn, constraints owned by a single declaration, repository-wide constraints, and turns that resemble constraints but are scoped to the current task. Synthetic messages span clean, casual and noisy registers, sampled independently of the label so that writing style carries no information about the target. Approximately 72% of capture examples are labelled as containing nothing to retain.

The model emits one line per entry or the token NONE. Each entry specifies a kind, a source (user or agent), an anchor, and a body, for example MEMORY | kind=law | source=HUMAN | anchor=general | Env vars get read in exactly one place, config/env.js. The body is copied from the user’s statement rather than composed. A small model is considerably more reliable at locating and extracting a span than at generating a faithful paraphrase, and copying prevents the model from attributing to the user content the user did not provide.

4.3 Training

We fine tune Qwen3-1.7B with LoRA (rank 16, alpha 32) for two epochs on a single NVIDIA T4, computing the loss only on the target. The model is used in its non-thinking mode, and inference reproduces the training template exactly. The quantised model (Q4_K_M) occupies 1.3 GB.

5. Offline evaluation

Table 1 reports performance on 260 held-out capture examples, evaluated with the full judgment pipeline and its deterministic checks. Field-level accuracies are computed over examples in which the predicted and reference entry counts agree. The held-out examples come from the same transcript corpus and synthetic process as the training set, so these figures describe in-distribution performance; the cross-session benchmark is the more demanding test. Table 2 summarises runtime cost on a laptop-class machine. Because the hook only enqueues, its latency is the only overhead visible to the user.

6. Cross-session benchmark

6.1 Codebases and constraints

We use two codebases in different languages: a production Node.js backend service of approximately 2,400 files, and a widely used open-source Python web framework of approximately 110 source files. For each we wrote five constraints that cannot be inferred from the code: in total four laws, two hazards, two corrections and two rituals. Each constraint deliberately contradicts the prevailing pattern in its codebase. In the Node.js service, for example, environment variables are read directly in 111 files and console logging appears at 393 call sites; in the Python framework one path library is used 35 times for every 3 uses of the alternative. An agent without memory that imitates surrounding code will therefore tend to violate the constraint. We regard this as the situation in which memory is most consequential rather than as a representative sample of all constraints.

6.2 Protocol

Each constraint is stated in a first session: a real agent session in which the user requests a small unrelated change and mentions the constraint in passing. Phrasing varies across constraints in ways typical of developer speech (“one standing rule going forward”, “fyi, we’re moving off”, “heads up, this bit us before”, “remember this for the future”). Two additional first sessions contain directive language but no standing constraint, and serve to detect spurious recording.

Second sessions are fifteen tasks, ten on the Node.js service and five on the Python framework, each exercising two to four constraints without mentioning them. The agent is Claude Code with Claude Sonnet 5 at medium effort, run non-interactively with a limit of 40 turns. Each session starts from a fresh copy of the repository. Dependency installation and web access are disabled.

6.3 Conditions

Four conditions differ only in who writes memory and how it is delivered.

• No memory. Each second session begins without access to anything recorded earlier.
• Built-in memory. Unmodified Claude Code. The agent’s memory directory persists between sessions, and any instruction files it writes are carried into later sessions as they would be in a working repository.
• 54Humans. The agent has no means of writing memory. After each first session, 54Humans judges the transcript and writes to the memory store, which injects repository-scope entries at the start of each second session, up to eight entries within 2,000 characters.
• 54Humans via built-in memory. This condition separates the writer from the delivery channel. The entries 54Humans wrote are placed, verbatim, in the agent’s built-in memory directory, in the format the agent uses for its own entries, and second sessions run as in the built-in memory condition with no store and no injection.

6.4 Scoring and analysis

A script examines the lines added by the agent in each second session. A constraint is scored only if the task exercised it, determined by the presence of the relevant behaviour in the added code; it is then scored as complied with or violated. No language model is involved in scoring, and every violation was additionally checked by hand. Because pairs within a session are not independent, we report Wilson 95% confidence intervals and Fisher’s exact test at the pair level together with two coarser analyses: the proportion of sessions in which every exercised constraint was respected, and a paired comparison over tasks. The four conditions comprise 90 agent sessions (median 11 turns) at a total cost of USD 23.65.

7. Results

7.1 Compliance

Built-in memory roughly doubles compliance relative to no memory (29.6% to 63.0%, p = 0.0012), and 54Humans raises it further to 94.5% (p = 9.5×10⁻⁵ against built-in memory). The same ordering holds at the session level: every exercised constraint was respected in 16 of 18 second sessions under 54Humans, 9 of 15 under 54Humans via built-in memory, 3 of 15 under built-in memory, and none of 18 without memory. In a paired comparison over the fifteen tasks, 54Humans achieved higher compliance than built-in memory on 10, equal on 5, and lower on none (two-sided sign test, p = 0.002). In repeated runs, all 19 constraint outcomes matched the first run.

One constraint, requiring use of a shared logger, was respected in 10 of 11 cases even without memory, because code surrounding the relevant call sites already used the logger. Excluding it, compliance is 14.0% (6/43) without memory, 52.8% (19/36) with built-in memory, 83.3% (30/36) with 54Humans via built-in memory, and 92.9% (39/42) with 54Humans.

7.2 Writer or channel?

54Humans differs from the built-in memory condition in two respects: who writes the entries, and how they reach the next session. The fourth condition holds the channel fixed. With 54Humans’ entries delivered through the agent’s own memory directory, compliance is 87.0%, against 63.0% when the agent writes that directory itself (p = 0.015). Over the fifteen tasks, 54Humans’ entries produced higher compliance than the agent’s own on 9, equal on 6, and lower on none (p = 0.004). Moving the same entries to the memory store raises compliance further to 94.5%, a difference that is not significant at this sample size (p = 0.29).

Most of the effect lies in what is recorded, not in how it is delivered.

The remaining gap is concentrated in a single constraint. Four of the six violations in the via-built-in condition concern the rule on environment variables, whose entry reads as a description (“Env vars get read in exactly one place, config/env.js.”) rather than an instruction. Presented as one note among the agent’s own memories it was read and not applied; presented by the memory store, under a header stating that the entries are rules given by the developer, it was followed in all nine sessions that exercised it.

7.3 An example

In an earlier session the developer stated two constraints relevant to one task: that new code must use the built-in fetch rather than axios, introduced with “fyi”, and that environment variables are read only in a single configuration module. The task asks for a notifier that posts to a Slack webhook whose URL is given by an environment variable.

Without memory the agent reproduced the prevailing patterns of the codebase. With its built-in memory it followed the one relevant convention it had recorded, routing its messages through the shared logger, and violated the two it had not. With 54Humans every relevant constraint was present at the start of the session, and the agent added the variable to the configuration module, imported it, and used fetch. The agent follows what it is given, and the conditions differ in what it is given.

7.4 Compliance by kind of constraint

Rituals are the most dependent on memory: without it they were never performed, since nothing in the code indicates that a given change entails an additional step. All three memory conditions performed well on rituals, which reflects that both writers recorded them. The largest differences between the writers arise for laws, hazards and corrections.

7.5 The role of delivery

For each exercised pair we determined from the session transcript whether the constraint was present in the agent’s context, through the memory store’s injection, the agent’s memory directory, or an instruction file. Pooled over the four conditions, delivered constraints were followed in 95.0% of cases (114/120, 95% CI [89.5, 97.7]) and undelivered constraints in 28.4% (23/81, [19.7, 39.0]); Fisher’s exact test gives p = 1.4×10⁻²⁴.

Once delivered, a constraint was almost always respected. The dominant failure is not disregard of stated constraints but their absence from context.

The undelivered constraints that were nonetheless respected were predominantly cases in which the task itself favoured compliant code. In each condition the compliance rate sits close to the delivery rate: 0% of constraints were delivered without memory, 50% with built-in memory, and 96% in both 54Humans conditions.

7.6 What each writer recorded

The agent’s own entries were of high quality. Each restated the constraint, noted that the user had requested it, and described how to apply it, and the four were followed in 22 of the 23 later cases that exercised them. The difference between the conditions lies in coverage rather than in the quality of individual entries. 54Humans produced one entry that did not correspond to a constraint, derived from a task description phrased as a procedure; it was anchored to a single file and was not injected into any later session.

8. Discussion: where memory is lost

8.1 Selective recording by the agent

All four constraints the agent recorded on its own were phrased explicitly as durable rules (“a standing rule for this codebase”, “remember this for the future”). Of the six it did not record, five were phrased incidentally (“fyi, we’re moving off”, “heads up, this bit us before”, “also, going forward”); the sixth was introduced as “one standing rule going forward” but embedded in the description of the task. An agent engaged in a task appears to record a constraint when the user clearly signals that it should be recorded and the statement stands apart from the work, and otherwise treats it as part of the current task. 54Humans, whose only function is to make this judgment, recorded five of these six.

We also evaluated a variant in which the agent was given an explicit tool for writing to the same memory store used by 54Humans, with a description instructing the agent to call it whenever the developer states a rule, corrects it, or asks it to remember something. The variant repeated the full protocol with the store delivering entries exactly as in the 54Humans condition. Compliance was 46.7% (21/45). When the agent used the tool, it frequently anchored an entry to a directory or to a file that did not yet exist; the store correctly declined to treat such entries as verified and did not deliver them, while reporting the write as successful to the agent.

8.2 Delivered but not followed

Six delivered constraints were violated across all conditions. In the 54Humans condition the agent raised a bare RuntimeError despite a delivered prohibition. In the built-in memory condition it once skipped the endpoint-list ritual it had itself recorded. The remaining four were the environment-variable rule in the via-built-in condition. The contrast with the memory store, where the identical entry was always followed, suggests that the framing of delivered memory matters for entries phrased as descriptions: an agent that is told a list of entries are instructions from the developer treats them as such, while an agent that finds the same sentence among its own notes may treat it as context. Recording entries in imperative form would make them robust to either channel.

9. Limits and open questions

What follows is where the evidence stops: the places where the design or the experiment is narrower than the claims it might seem to support.

Ten constraints, written by us. Each codebase received five constraints of our own choosing, each chosen to contradict the prevailing pattern in the code. This isolates the contribution of memory, but it also means the absolute figures describe that situation rather than the typical constraint a developer states. A constraint that agrees with the surrounding code would be followed without memory, and a benchmark built from those would show a smaller effect. Whether the gap between writers persists on constraints drawn from real sessions is the first thing we would test next.

Small numbers. The four conditions rest on ninety sessions and fifteen tasks. The effects are large and survive the session-level and task-paired analyses, but the intervals are wide, and the difference between the two 54Humans conditions cannot be resolved at this size. The ordering of the conditions is well supported; the precise magnitudes are not.

One agent. Every session used Claude Code with a single model at a single effort level. A different agent, or the same agent with a different memory policy, might record more. The finding that delivered constraints are followed is likely to hold more widely, since it concerns instruction following rather than memory; the finding that agents record too little is specific to what we measured.

A single interval. The benchmark measures one gap between a first session and a later one. It does not test a store that grows over months, entries that become stale as the code changes, or the cost of delivering many entries at once. The memory layer retires entries whose subject disappears, but how a store written by 54Humans behaves at that scale is an open question and, we think, the more important one.

Scoring by pattern. Compliance is determined by pattern checks over the lines an agent added, and delivery by inspecting transcripts for the constraint’s text. Every violation was verified by hand, and one error in the delivery check was found and corrected in that process, but an unusual compliant idiom could still be misclassified, and delivery through a channel we did not anticipate would be missed.

10. Related work

Memory for language model agents. MemGPT treats the context window as a managed memory hierarchy controlled by the model. Generative Agents maintain a memory stream with periodic reflection, and Reflexion stores verbal lessons from failed attempts. Mem0 and A-MEM provide memory layers that extract and organise information from conversations, generally using a large model for extraction. Zep maintains a temporal knowledge graph and uses a model to decide when a new fact supersedes an old one, and EA-Graph anchors a coding agent’s verification claims to the artifacts that established them. In these systems the decision of what to store is made by the acting model or by a large auxiliary model. 54Humans assigns this decision to a dedicated small model and removes write access from the acting agent.

Why accumulation is not free. Delivering stored material to an agent has a cost that grows with the store. Instruction following degrades as the number of instructions rises, and agent instruction files more than triple in length over their lifetime while their oldest entries become the hardest to remove. Both bear on the design choices here: 54Humans records rarely, extracts rather than elaborates, and relies on the memory layer’s anchoring to retire entries whose subject is gone.

Instructions for coding agents. Instruction files such as CLAUDE.md and AGENTS.md, and the memory directory maintained by Claude Code, are the prevailing mechanism for persisting project knowledge. Our results suggest that these mechanisms are effective when populated, and that population, rather than consumption, is the limiting step.

11. Conclusion

We examined how a coding agent’s knowledge of project constraints can persist across sessions, and argued that the task of deciding what to record should be separated from the task the agent is performing. 54Humans, a 1.7B-parameter local model with exclusive write access to the agent’s memory, recorded nine of ten constraints stated during unrelated work, recorded nothing from turns that did not warrant it, and raised compliance in later sessions from 63.0% with the agent’s built-in memory to 94.5%. Delivered through the agent’s own memory, the same entries still reached 87.0%. In the terms of our earlier work, 54Humans lowers the authorship floor from the capture side: what a developer says in passing is recorded without anyone deciding to author it.

Across conditions, compliance tracked delivery closely, which suggests that further gains in this setting will come from improving what is recorded and how it is delivered. Directions for future work include relevance-based delivery for large stores, training data drawn from many users, and evaluation over longer horizons in which stored constraints change or become obsolete.

References

1. [1]K. Chakrabarti. Why Does CLAUDE.md Keep Growing? Catastrophic Remembering in Agentic Coding. arXiv preprint arXiv:2608.11095, 2026.
2. [2]P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv preprint arXiv:2504.19413, 2025.
3. [3]H.-J. Hsu, C.-J. Chi, and H. Everett. EA-Graph: Artifact-Anchored Verification Memory for Coding Agents under Upstream Drift. arXiv preprint arXiv:2608.04278, 2026.
4. [4]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR), 2022.
5. [5]D. Jaroslawicz, B. Whiting, P. Shah, and K. Maamari. How Many Instructions Can LLMs Follow at Once? arXiv preprint arXiv:2507.11538, 2025.
6. [6]C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In International Conference on Learning Representations (ICLR), 2024.
7. [7]A. Katakam. The Authorship Floor: Why Project Knowledge Is Lost at Both Ends, and What Anchoring It to Code Recovers. 2026.
8. [8]C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez. MemGPT: Towards LLMs as Operating Systems. arXiv preprint arXiv:2310.08560, 2023.
9. [9]J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), pages 2:1-2:22, 2023.
10. [10]Qwen Team. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388, 2025.
11. [11]P. Rasmussen, P. Paliychuk, T. Beauvais, J. Ryan, and D. Chalef. Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv preprint arXiv:2501.13956, 2025.
12. [12]N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language Agents with Verbal Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
13. [13]W. Xu, K. Mei, H. Gao, J. Tan, Z. Liang, and Y. Zhang. A-MEM: Agentic Memory for LLM Agents. arXiv preprint arXiv:2502.12110, 2025.