Who Decides What's Worth Remembering? Notes in the Margin of a Memory Paper

I spent a flight reading Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents (Wu et al., Meta AI, July 2026), and by the time we landed the margins were fuller than the abstract. Not because the paper is bad — it isn't. It names a real problem cleanly, runs honest ablations, and reports gains I believe. But I kept disagreeing with the shape of the answer, and the disagreements turned out to rhyme with two years of arguments we've had building memory for Otto, our AI travel agent.

So this is not a rebuttal. It's a set of questions I'd genuinely like the field to argue about, written by someone who has made most of these mistakes already.

What the paper proposes

The core observation is sharp and correct. In a long trajectory, a fact that should keep constraining behavior — a requirement stated early, a command that already failed, an open subgoal — gets buried in the context window and quietly stops influencing the next action. The authors call this behavioral state decay, and they're right that it's distinct from "can we store and retrieve this." The information is often still in the context; it just no longer steers.

Their fix: run a second memory agent alongside the unmodified action agent. On a fixed schedule it maintains a structured memory bank — a private status field, plus knowledge and procedural entries — through a small set of tool calls, and then decides whether to inject a targeted reminder into the action agent's next call, or stay silent. The headline result is that selective intervention beats both always-on injection and passive retrieval (Mem0-style), improving pass@1 on Terminal-Bench 2.0 and τ²-Bench for both weak and strong action models. They even show the intervention policy is partly learnable by fine-tuning a smaller open model.

Good problem, careful evaluation. Here's where I start scribbling.

The question underneath everything

Somewhere over the Rockies I realized my objections weren't four separate complaints. They were one question wearing different outfits.

Memory for agents is usually framed as storage and retrieval — what to keep, how to index it, how to fetch it back. But that's not the hard part. The hard part is a decision: who gets to decide that a piece of past state is relevant to the action being taken right now? The memory system, watching from outside? Or the acting agent, which actually holds the goal?

Once I read the paper through that lens, choices that looked unrelated — the fixed schema, the separate memory agent, the selective injection, even the latency argument — turned out to be consequences of the same bet about where the authority to judge relevance should live. That's the thread I'll pull.

What should memory represent?

Start with the memory bank. It's a hand-designed ontology: three components — status, knowledge, procedural — with entries in a "compact tagged format" (environment facts, paths, task facts, bugs, performance observations). Someone decided in advance that this is the right vocabulary for an agent's execution state.

It's tempting to wave the Bitter Lesson at that and walk away — Sutton's warning that the methods which scale with computation beat the ones that build in how we think a problem should be structured. But that's too blunt an instrument, and a sharp reader will bat it away: the Bitter Lesson says don't hand-engineer the intelligence, not don't give an intelligent system structured state. Context windows have structure. Tool APIs have schemas. Databases have schemas. Structure isn't the sin.

The precise objection is narrower, and the paper hands it to me itself: they learn when to intervene, but fix in advance the vocabulary the intervention is built from. The intervention policy is trained with SFT and GRPO — the learned, scalable half — while the schema stays fixed and human-authored, and the action agent is left untouched. So the real question isn't "is any ontology a Bitter-Lesson violation." It's what deserves to be learned, and what deserves to stay architectural? Draw that line in the wrong place and you get a scaffold you'll be embarrassed by in two years — the memory equivalent of hand-tuned image features right before convolutional nets ate them. I don't know that status/knowledge/procedural is the wrong line. I know it was drawn by hand, and that the paper never asks whether it should have been.

Can memory know what matters without owning the goal?

This is the center of the essay.

The memory agent's entire job is to decide what execution state is "decision-relevant" and worth reactivating. But relevant to what? Relevance is defined by the action agent's goal and its live understanding of the task. The moment you split memory into a separate agent with its own context, you've created a second model that has to stay aligned with the first about the purpose of the task and about what "matters" right now.

That's an alignment problem you've manufactured inside your own system. If the memory agent doesn't fully share the actor's goal, its interventions are educated guesses about someone else's intent. And if it does share the goal well enough to judge relevance correctly — why is it a separate agent at all, rather than a faculty of the one that already holds the goal?

That's the dilemma the paper's framing steps around: it assumes the observer knows better than the actor when something matters. Sometimes it will. But that's an assumption, not a result — and it's the one I trust least.

Latency isn't the cost I'm most worried about

The paper motivates selectivity partly on latency — surfacing too much memory "adds latency." Longer input isn't free, and I won't pretend otherwise: it touches prefill, cache behavior, memory bandwidth, and, depending on your provider, price. How much it matters relative to decode depends entirely on your serving setup.

But that's not the failure mode I lose sleep over. The one I do is attention dilution: bury the load-bearing constraint among forty plausible-looking memories and the model loses it — the "lost in the middle" effect the paper itself cites. That reframes the design goal. If the enemy were milliseconds, the fix would be "inject less." If the enemy is dilution, the fix is "make the one thing that matters unmissable at the moment of decision." Different problems, different solutions — and I think the second is the real one.

What Otto taught us

Up to here this is a paper critique. Here's where the scars come in.

We've lived this arc. We started with RAG two years ago — then Zapier wiring, Mem0, a few other knowledge bases — and hit a wall that wasn't about retrieval quality: the frameworks weren't built to treat memory as first-class, and context windows weren't yet big enough to force the issue. So we settled on a three-layer architecture — session (per trip), user (per traveler), and global (company policy plus distilled cross-traveler knowledge). It held for a long stretch.

It's straining now, and the examples are the point. New modes — delegate booking and group travel — broke the tidy "one traveler, one trip" assumption the layering was built on. Passive context — hotel, flight, and local knowledge we distill at a near-real-time cadence — is a firehose none of the layers were sized for. And a bigger action surface — calendar, booking records, car legs, not just flights and hotels — means far more state that could matter dozens of turns later. Sensible abstractions decay as the product grows past the assumptions they were drawn under.

So we're exploring two moves: push durable memory from the session layer into the user layer, leaving the session to hold only the live conversation; and treat retrieval as an explicit tool the agent calls, trading latency for context-window space. That second move is the latency-for-space trade I just questioned — except now I'm paying it knowingly, to keep the window clean enough that the constraint that matters still steers the next decision. Dilution, not milliseconds.

Make it concrete with one preference — an aisle seat.

Both architectures remember the aisle seat. Storage was never the question. The question is which component recognizes that seat preference has become decision-relevant right now — the observer watching from outside, or the actor standing at the seat map. That one example is the whole argument, and our bet is that the actor, holding the goal, is better placed to make the call.

The uncomfortable counterargument

Except the paper's best result is a loaded gun pointed straight at our bet.

Selective intervention beats always-on injection and beats passive retrieval. For a pull-based camp like ours, that's a warning, not a vindication — because pull has a failure mode push doesn't. Pull requires the actor to know that it has forgotten something. But behavioral state decay is precisely the condition where the old constraint no longer influences the actor's next action. If the constraint has already gone quiet in the actor's head, what makes us think the actor will decide to call the tool that retrieves it?

So neither obvious answer works cleanly. Push means the observer may not understand relevance, because it doesn't own the goal. Pull means the actor may not seek the memory, precisely because the relevant state has already decayed. That's not a gotcha against the paper — it's the genuine unsolved problem sitting under both designs. And it points somewhere more interesting than "pick a side": learned retrieval triggers inside the actor, triggers keyed to action type rather than a fixed clock, shared latent state between the two roles, hybrids that push a nudge and let the actor pull the detail. I don't know which wins. I know the trade-off is real.

The questions worth arguing about

If you're building agent memory in production, these are the three I'd put on the whiteboard:

I don't have clean answers, which is the point. If you've watched a layered memory architecture buckle the way ours is, I'd love to compare scars.

(These notes started as scrawls on a GBTA Engine 2026 notepad, which tells you something about where my head is: memory isn't an abstract ML problem for us, it's why a traveler's aisle-seat rule survives to leg four.)

Reference: Wu, Zhang, Zhou, Wang, Peng, Li, Fan, Zhao. "Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents." Meta AI, arXiv:2607.08716, July 2026.