Why AI Agent Design Is Changing Faster Than Most Teams Realize
Three threads worth acting on this week: KV cache reuse as the cost lever everyone keeps rediscovering, agent skills replacing the tool call as the unit of agent design, and performance forecasting becoming an exercise we can defend with numbers. The fourth, RL credit assignment, is the one we're watching but not acting on yet.
Most weeks the radar finds nothing worth writing up. This week four papers landed on the same Tuesday and one of our internal Slack threads grew past ten messages, so we are writing the post instead.
Three of them changed something already on our roadmap, and the fourth changed what we expect the roadmap to look like a year from now. Worth pulling out in order.
1. KV cache is the cost lever everyone keeps rediscovering
Four days in a row, papers and launches converged on the same observation: the cache is where production economics live.
KVBoost (Show HN) claims 5–48× faster TTFT through chunk-level KV cache reuse in HuggingFace. The same week, KVServe appeared on the radar three days running — service-aware compression of the KV cache for disaggregated serving, where prefill and decode run on different machines and the cache crosses a network. DeepSeek's reasonix was the third release that treated the cache hit rate as a first-class cost input, not an accident. From the hardware floor, a Hacker News thread on the 25th: memory is now nearly two-thirds of AI chip component cost.
These are not the same paper. The thread underneath: at four different layers of the stack, the field is internalising the same observation. Input tokens repeated across turns are where the bill actually lives, and the cache is the cheapest tool for cutting them.
This is not new. Anthropic shipped prompt caching in 2024. The DeepSeek-V3 paper from late 2024 organised its entire inference path around the cache. What is new is the granularity. KVBoost is chunked, not prefix-locked, which means the cache can hit on text that appears mid-message: a quoted document section, a tool result that recurs across sessions, a piece of retrieved context. KVServe takes the same observation and pushes it across the wire: compress the cache to a size where the network is no longer the bottleneck. Memory pricing closes the loop. Cache reuse used to be a percentage optimisation. Now it is what separates a workload that runs at margin from a workload that runs at cost.
What it looks like inside JARVIS
Our JARVIS routing layer runs explicit cache_control breakpoints. The structure is conceptually simple. The prompt is built in layers, each layer gets a breakpoint based on how often it changes:
# Pseudocode for clarity — actual construction lives in jarvis/prompt/builder.py
prompt = [
# Layer 1: stable instructions, change weekly at most
{
"role": "system",
"content": [{
"type": "text",
"text": SYSTEM_PROMPT,
"cache_control": {"type": "ephemeral", "ttl": "1h"}
}]
},
# Layer 2: client context, change per-engagement
{
"role": "user",
"content": [{
"type": "text",
"text": client_context,
"cache_control": {"type": "ephemeral", "ttl": "1h"}
}]
},
# Layer 3: recent conversation, change per-turn
{
"role": "user",
"content": [{
"type": "text",
"text": recent_turns,
"cache_control": {"type": "ephemeral", "ttl": "5m"}
}]
},
# Layer 4: current turn, no breakpoint
{
"role": "user",
"content": current_user_message
}
]
The 1-hour breakpoint sits on text that is stable across an entire engagement: the system prompt, the client's reference architecture, the schema definitions JARVIS works against. The 5-minute breakpoint sits on the conversational tail. The current turn rides naked. On a representative multi-turn production session, this geometry yields a measured 68% cache hit rate on input tokens.
Sixty-eight percent is not a benchmark-beating number. It is a measured one, with deploy-blocking eval gates on regressions against it. The number itself will rise as the chunk-level techniques in KVBoost find their way into hosted APIs and we no longer have to align our breakpoints with prefix boundaries.
What KVServe changes
The disaggregation angle in KVServe is where this gets interesting for our move off single-tenant hosted infrastructure. Today JARVIS runs against hosted APIs: Anthropic, Together, Groq. The economics of that path are the economics of the cache hit rate against the provider's billing structure. The multi-tenant deployment is on the roadmap; a venue-operator-class client will eventually require it for data-residency reasons. When JARVIS moves there, the question stops being "how often does the cache hit" and starts being "where does the cache live and what does it cost to move it."
KVServe's answer is that you compress aggressively, you make the compression service-aware (different downstream tasks tolerate different fidelity), and you accept a quality envelope rather than chasing lossless transfer. We are not building this. We are reading the paper and updating the architectural runbook for what we expect to build, in roughly Q3.
What changed this week: KVServe moved from "interesting paper" to "load-bearing reference" in the runbook entry for the multi-tenant move, on the back of the memory-cost thread. The move itself is now closer to the front of the queue than it was on Monday.
2. Agent skills are quietly becoming the unit of work
Maestro appeared twice this week: reinforcement learning to orchestrate hierarchical model-skill ensembles, where the orchestrator learns which skill to invoke and which model to invoke it on. SkillOpt: executive strategy for self-evolving agent skills, where skills accumulate execution feedback and improve themselves against it. HINT-SD: targeted hindsight self-distillation for long-horizon agents, which is how skills get better at multi-step problems without exploding their training cost. From Raw Experience to Skill Consumption: a systematic study of how model-generated skills get reused, decay, and need pruning. ACC: compiling agent trajectories into training data for long-context models, which is how skills end up as competence in the base model rather than scaffolding around it. GenEvolve and ClinSeekAgent: domain-specific orchestration in image generation and clinical reasoning, which are the early production demonstrations.
Two YC launches landed the same week: Runtime ships sandboxed coding agents for teams, and Superset ships an IDE for the agents era. Both are organised around agents as the development primitive rather than as something developers occasionally invoke.
The thread underneath is a shift in granularity that is easy to miss because each individual paper is small.
A year ago the unit of agent design was the prompt. You wrote a long prompt with instructions, examples, and a tool list, and you hoped. Six months ago the unit was the tool call: the agent's job was to pick the right tool, invoke it with the right arguments, and react to the result. Tools were stateless functions described by JSON Schema and discovered via something like MCP.
This week the literature is converging on the skill. A skill is a named, evaluable, versioned unit of agent capability. It carries its own eval set and a promotion path. Sometimes the skill itself is a learned policy, not a switch statement. The orchestrator routes to skills, not just to tools.
Maestro is the strongest paper of the cluster because it is the one that admits the orchestration problem is itself a learning problem. You do not write the routing logic in YAML. You train it on production traces and let it discover that this kind of question routes to that kind of model with this kind of context window. The whole orchestration layer becomes a policy.
What it looks like inside JARVIS
Today JARVIS's tool registry is structurally MCP-style. Each capability is a YAML entry that declares its inputs, its outputs, its evaluation set, and its handler:
# tools/extract_invoice_lines.yaml
name: extract_invoice_lines
version: 0.3.1
description: |
Extract line items from a Slovak invoice PDF.
Returns structured items with quantity, unit, price, DPH rate.
input_schema:
type: object
properties:
pdf_path: {type: string}
expected_currency: {type: string, default: "EUR"}
required: [pdf_path]
output_schema:
$ref: schemas/invoice_lines.json
handler: jarvis.tools.invoice.extract_lines
eval_set: evals/invoice_lines_sk_v2.jsonl
eval_baseline:
field_accuracy: 0.94
line_recall: 0.91
deprecates: []
The eval_set field is load-bearing. A tool that does not declare an eval set cannot be promoted past the staging registry. The eval_baseline locks the floor — if the next version's eval drops below baseline, deploy is blocked. This is the same discipline we apply to model upgrades, applied one layer down.
What the literature this week proposes: let the handler itself be a learned policy on the subset that is mostly decisions (which model to invoke, which sub-tool to chain, which retrieval strategy to use) rather than mostly computation. We have not done that yet. The capability to do it requires a training loop we do not yet operate. ADR-015 is on the table to draft next week.
What we expect to change
ADR-015's working title is Skills as first-class artefacts. The proposal, in short:
- Versioned separately from the agent that invokes them.
- Declare their training data and training method, not just their handler.
- Carry a confidence score, not just an eval baseline. Below threshold, the orchestrator refuses to invoke.
- Promoted on the same gate as a model upgrade: eval score plus production trace quality plus human review.
This is not novel. It is roughly what Anthropic has done with the SKILL.md pattern in Claude Code, what OpenAI is gesturing at with their newer Assistants API, and what Maestro proposes mathematically. The novel part for us is making the same structural commitment internally: skills are not configuration; they are deployable artefacts with their own promotion path. The work over the next quarter is operational: building the training loop, building the promotion gate, building the rollback path when a learned skill regresses against its predecessor.
3. Performance forecasting is becoming tractable
Two papers worth reading together.
Forecasting Downstream Performance of LLMs With Proxy Metrics proposes a small, cheap-to-compute set of signals that correlate with full-evaluation performance on specific downstream tasks. The point is operational. Today we run a 240-instance eval against three candidate models to pick a routing target. That costs money and an hour of wall time per routing-decision class. The proxy lets us predict instead.
LLMs as Noisy Channels reframes scaling laws through Shannon information theory. The model is a noisy channel; training is the process of matching channel capacity to the complexity of the data being transmitted. The reframing is interesting because it explains things that the standard scaling-law literature describes as anomalies. Why certain quantisations break a model's reasoning while others do not. Why fine-tuning sometimes degrades capabilities in patterns that look chaotic. Why the relationship between parameter count and downstream performance is not monotonic on every task. Under the noisy-channel view, these are not anomalies. They are predictable consequences of capacity mismatch.
The cost-projection conversation Sebrona has with every client (what will this workload cost at scale, on which model) has been a vibes exercise for two years. These two papers are why it stops being one. It is now an exercise we can defend with numbers.
It is now an exercise we can defend with numbers.
What we did
The routing decision log already records, for every routing decision, the candidate models, the selected model, the input token count, the output token count, the cache hit rate, the latency, and the downstream evaluation score if one is available. We added one column this week: the proxy-metric prediction at the moment of routing.
A row looks roughly like this:
- ts
- 2026-05-23T14:22:09Z
- session_id
- jrv_01H4Y…
- task_class
- extraction_sk_invoice
- candidates
- [ opus-4-7, sonnet-4-6, mistral-small-3.2 ]
- selected
- sonnet-4-6
- selection_basis
- cost_quality_frontier
- proxy_metric
- 0.872 new
- proxy_metric_method
- ppl_calibrated_v1
- input_tokens
- 4,820
- output_tokens
- 612
- cache_hits
- 3,108
- cache_misses
- 1,712
- latency_ms
- 1,840
- downstream_eval_score
- null — filled when nightly eval lands
The proxy metric is a calibrated perplexity score against a 50-instance held-out set for the task class. We are not yet using it as a routing input. We are recording it alongside the actual outcome, on every routing decision, so that within a month we will have a calibration curve we can show clients. The curve will say: for tasks of this shape, the proxy metric predicts downstream performance within X percentage points, Y% of the time. Within a quarter, if the calibration holds, the prediction becomes a routing input. The orchestrator can then refuse to route to a model whose predicted performance falls below threshold, without paying for the full eval to confirm.
Measure first, predict second, automate third. The two papers cited above are why "predict" is now defensible without the full eval.
The noisy-channel framing matters even before the calibration curve is built. It tells us where to expect the proxy to fail. Tasks near the channel-capacity boundary of a model show worse proxy-to-outcome correlation than tasks comfortably inside the channel. The boundary cases are high-complexity, long-context, multi-step reasoning that compresses poorly under the model's representations. We expect the calibration curve to be tight for routine extraction and loose for novel reasoning. That is itself useful: it tells us which task classes can be cheaply routed by prediction and which require the full eval.
4. RL credit assignment — watching, not acting
We are not yet acting on the fourth thread. A consultancy that pretends to act on everything it reads becomes unreadable within six months.
DelTA appeared three days running: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards. From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning appeared on the 22nd. Unsupervised Process Reward Models appeared on the 23rd and 24th. Together they describe a quiet methodological shift in how reasoning models are trained: away from sparse outcome rewards on long reasoning chains, toward dense process rewards where each step of the reasoning gets credit proportional to its contribution to the final answer.
DelTA specifically discriminates which tokens in a reasoning trace were responsible for the correct outcome: the credit-assignment problem applied at token granularity, with the verification of the final answer as the supervisory signal. The curriculum paper decomposes hard reasoning problems into verifiable subproblems where credit can be assigned locally, then schedules the curriculum so the model learns the subproblems before it tries to compose them. The unsupervised PRM work removes the requirement that the per-step rewards be human-labelled, which is the expensive bottleneck that has held process reward models back from scale.
The reason this matters is that the reasoning models we will route to in late 2026 and early 2027 will have been trained this way. The cost-quality frontier shifts when reasoning gets cheaper to train well. Some skills that work poorly on today's models because the chain breaks halfway through start working: invoice arbitration, long-form contract review, multi-step engineering reasoning, the harder cases in revízna správa generation. The orchestrator's job shifts with them. Some of what we route to Opus 4.7 today (because it is the only model that holds the chain together) will route to a Sonnet-class model when the next Anthropic release ships, expected Q4. The price drops by more than half on that move.
We are not acting on this thread. No in-house PRMs, no fine-tuning. What is changing is the rolling twelve-month projection of the cost-quality frontier that sits in the back of every client proposal. The line moved this week, and we moved with it.
This is the thread that matters most over the next year, even though it changes nothing this week. We separate what we ship now from what we expect to ship in twelve months because that is what clients pay us for.
What we publish, and why
We publish field notes because we read the literature anyway, and the only question is whether the reading surfaces.
Our editorial position is that an architecture practice's job is to translate this week's research, releases, and launches into next month's deployment decisions, and to say so explicitly. Most write-ups stop at "interesting paper." Ours stop at "ADR opened" or "runbook updated" or "the projection moved." That last move is the one that distinguishes a practice from a newsletter.
The format is intended to run weekly. Each post pulls two to four threads from the JARVIS radar, ranks them by operational relevance, and ends each thread with the concrete consequence: code change, ADR draft, projection revision, or honest abstention. We expect roughly one post a month to include a thread we are explicitly not acting on. Those threads are tagged "watching" so you can grep for them later.
Errors welcome. If a citation, a result, or a claim is wrong here, write to info@sebrona.com. Corrections land in public on this post.
Reading
Items from the JARVIS radar this post draws from, in order of appearance:
- KVBoostChunk-level KV cache reuse for HuggingFace, 5–48× faster TTFT.
- KVServeService-aware KV cache compression for communication-efficient disaggregated LLM serving.
- DeepSeek reasonixNative agentic coding framework organised around prompt caching and cost.
- Memory has grown to nearly two-thirds of AI chip component costs
- MaestroReinforcement learning to orchestrate hierarchical model-skill ensembles.
- SkillOptExecutive strategy for self-evolving agent skills.
- HINT-SDTargeted hindsight self-distillation for long-horizon agents.
- From Raw Experience to Skill ConsumptionSystematic study of model-generated agent skill reuse.
- ACCCompiling agent trajectories for long-context training.
- GenEvolveSelf-evolving image generation agents via tool-orchestrated visual experience distillation.
- ClinSeekAgentAutomating multimodal evidence seeking for agentic clinical reasoning.
- Runtime (YC P26)Sandboxed coding agents for teams.
- Superset (YC P26)IDE for the agents era.
- Forecasting Downstream Performance of LLMs With Proxy Metrics
- LLMs as Noisy ChannelsA Shannon perspective on model capacity and scaling laws.
- DelTADiscriminative token credit assignment for reinforcement learning from verifiable rewards.
- From Reasoning Chains to Verifiable SubproblemsCurriculum reinforcement learning for LLM credit assignment.
- Unsupervised Process Reward Models
- Efficient Agentic Reasoning Through Self-Regulated Simulative Planning
- TerminalWorldBenchmarking agents on real-world terminal tasks.
Sebrona internal references
Recent entries from the ship log that ground the claims above. Full log at /changelog.
- 25 May 2026JARVIS tool-registry cache invalidation; hit rate held at 68% across the deploy.
- 14 May 2026JARVIS routing layer hits v2.
- 02 May 2026tool registry refactor.
- 18 Apr 2026prompt-cache plumbing made explicit.
- 11 Apr 2026cost ledger with Slovak DPH treatment.
- 09 May 2026Reference architecture v1.2 — at /tech. ADR-014 documents the orchestration-vs-integration layer re-cut behind it.
- 01 Jun 2026ADR-015 — skills as first-class artefacts. draft. Will publish at /tech when promoted.