Lecture 19 decided whether a stalled session's cache should continue to exist. Today decides where it should live, and the answer turns out to be settled by a single division that most deployments have never performed. A session's KV cache is bytes with a known rebuild cost, which makes it the rare cache whose miss penalty you can compute in advance — and comparing that penalty against a tier's bandwidth tells you, tier by tier, whether the cache belongs there. Everything from HBM down to a local NVMe drive passes; only the slow network fails. Then the lecture turns around and looks upward, at the client, because the context policy your assemble function implements decides what the server's cache is even able to reuse — and there the ranking from Lecture 4 inverts completely: the policy that submits the fewest tokens computes the most. By the end you should be able to place a session's state on a storage hierarchy, derive the bandwidth below which a tier stops being worth using, set a retention TTL from first principles, and explain why a context-management change you make as a client can quadruple a server's prefill bill without appearing on your own invoice.

Where this sits

Lecture 13 built a prefix cache and showed that identical prefixes have identical caches, which made sharing sound. Lecture 19 established that an agent session spends half its life holding a cache it is not using. Today those meet: the shared prefix and the parked session are the same bytes, and the question of where to put them is a storage-hierarchy question with an unusual property — the backstop tier is recompute, whose cost is known in advance and proportional to length. Lecture 13 noted this in one paragraph and computed one break-even at 4,096 tokens; today we take it to session scale, where the numbers move and the conclusions sharpen. Later in this same meeting, Lecture 21 multiplies the session by k and asks what happens when eight of them share a parent.

Instructor notes — Timing plan

75-minute class (Mon/Wed 11:15am–12:30pm, SEC 2.118). This plan is the standalone budget; on the day this subject shares the Nov 23 meeting with Lecture 19 (tool stalls) and Lecture 21 (multi-agent workloads), so read the row minutes as relative weights. The meeting is a student-led paper discussion, so the presenting group owns Continuum and the instructor holds only the §20.2 break-even and the §20.4 inversion; the discussion seeds after §20.8 are what the room is actually for, and seeds 1 and 3 are the two worth protecting.

Time	Segment	Notes
0–5	Framing; A5 due Dec 2	"The first subject asked whether the cache should exist. Now: where it lives — and then who ruined it."
5–12	§20.1 The session as a stateful object	Lifetime table. Fast; it is orientation.
12–24	§20.2 The fetch-versus-recompute line	Derive 2.94 GB/s live from Lecture 13's 4.2. The falling-with-length result is the payoff.
24–32	§20.3 The tier table	Read the ratios aloud. Land on "only the slow network loses."
32–55	§20.4 The compaction inversion	Centerpiece — never cut. Derive 7,700 / 32,000 / 11,200 on the board next to Lecture 4's 97,000 / 83,500 / 75,100. Both tables must be up together at the end.
55–63	§20.5 Retention and TTL	The p · C / f derivation; connect to Lecture 19's 21.3 s.
63–70	§20.6 Migration	43 ms versus 0.731 s, and 12× at 4K against 17× at 16K. Short.
70–75	§20.8 The invariant + close	"The KV cache can never be wrong, only absent." Sort the six mechanisms into costs time and costs correctness, then point at Lecture 21, next in the same meeting.

Reading-only, not scheduled: §20.7 (memory above the cache, MemGPT re-read) and the pricing block at the end of §20.4 — though if five minutes appear, the cached-input pricing table is the best possible use of them, because it explains why the client's incentives are misaligned.

If running long: compress §20.3 to three rows — HBM, DRAM, NVMe — and drop §20.6 to a sentence. Never cut §20.4; the inversion is the lecture, and it is the only place in the course where a Lecture 4 conclusion gets reversed by a Part II fact.

Learning objectives

By the end of this class you should be able to:

1. Characterize a session's state by lifetime, growth pattern, and access burstiness, and say which of those makes it unlike a chat request.
2. Derive the fetch-versus-recompute break-even bandwidth for a given model, context length, and attention layout, and explain why it falls as sessions get longer.
3. Place session state on a storage hierarchy and name the tiers that lose to recompute.
4. Compute, for a given client-side context policy, how many tokens a perfect prefix cache actually has to compute — and explain why the ranking differs from the submitted-token ranking.
5. Derive a retention TTL from a rebuild cost, an occupancy fraction, and a probability of return.
6. Decide between migrating a session and rebuilding it on the destination.

20.1 The session is a stateful object with a lifetime

Everything Part II built before Lecture 18 assumed the unit of state was a request. Put the two side by side and the mismatch is immediate.

Property	Chat request	Agent session
Lifetime	Seconds	Minutes to hours; sometimes days across resumptions
State growth	Fixed at admission, bounded by the output cap	Monotonic, unbounded, growing by a tool result of unknown size each step
Access pattern	Continuous until done	Bursty: 1.5 s of use, 1.5 s idle, twenty times
Ends when	The response completes	Nobody knows. The user may return in a second or never
Rebuild cost if lost	One prefill of a known prompt	0.731 s of GPU, and it grows with the session

The fourth row is the one that has no analogue anywhere else in the course. A chat request has a defined end; an agent session has an indeterminate one, and every retention policy has to make a decision under that indeterminacy. That is what makes retention a probability problem rather than a capacity problem, and §20.5 works it out.

The third row is Lecture 19's, restated: burstiness is what makes tiering worth doing at all. If a session used its cache continuously there would be nothing to move; the 50% duty cycle is the window in which a transfer can hide.

The second row is why admission is hard and, as it turns out in §20.4, why the client's context policy is a serving parameter.

Instructor notes

Minutes: 7. Board: The five-row table, chat column first, then agent column filled by the room. Ask the room: "Which row would you most like to change?" Usually "ends when." Ask what it would take: a client that declares the session closed — which is, again, free structural information the API has nowhere to put.

20.2 The fetch-versus-recompute line, at session scale

Lecture 13 made the observation that separates this cache from every classical one: a miss has a second recovery path. A page cache that misses must fetch; a KV cache that misses may fetch or recompute, and the recompute cost is known in advance and proportional to length. So the question "should this tier hold session state" is not a matter of taste. It is a division.

Fetching is worth it above that bandwidth and recomputing is worth it below. Lecture 13 evaluated it for a 4,096-token prefix. Redo it at session scale.

The break-even falls as the session gets longer, and the reason is worth saying out loud because it is the only place in the course where the quadratic term helps us. Bytes grow linearly in S; recompute grows superlinearly, because attention is quadratic. So the longer the session, the more expensive it is to rebuild relative to its size, and the slower a tier may be while still beating recompute. Long sessions are the easiest thing to tier, not the hardest — which is the opposite of the intuition that says big things are hard to move.

Two consequences follow immediately.

Architecture decides which tiers are available. GQA-8's 2.94 GB/s versus MHA's 11.7 GB/s is a 4× difference in how slow a tier may be. That is what puts NVMe in play for one and out of play for the other, and it is the same model-architecture decision that Lecture 2 credited with taking the batch ceiling from 29 to 116. One choice, three payoffs, at three different layers.

Compression moves the line rather than accepting it. CacheGen's position: if the tier you want is below the break-even, shrink the bytes. Halving the cache with a lossy encoding halves the required bandwidth, at a quality cost you have to measure — a pruning-and-quantization (Nov 9) tradeoff arriving in the storage layer.

Instructor notes

Minutes: 12. Board: The division, then the four numbers in a 2×2 (S by attention layout). Circle the falling column and ask why before explaining. Ask the room: "Prefill got 2× faster next generation. Which way does the break-even move?" Up — faster recompute makes fetching less attractive. Hardware improvement can remove a tier from play. Expect confusion: Students read "break-even bandwidth" as a property of the network. It is a property of the model and the accelerator; the network is what you compare against it.

20.3 The tier table, and where session state actually fits

Now place the tiers against 2.94 GB/s.

Tier	Bandwidth	Versus break-even	Time to move one 2.147 GB session
HBM (resident)	3,350 GB/s	1,140×	0.6 ms
NVLink-class peer GPU	900 GB/s	306×	2.4 ms
PCIe Gen5 ×16 to host DRAM	64 GB/s	21.8×	34 ms
400 Gb/s RDMA to a remote node	~50 GB/s	17×	43 ms
100 GbE	12.5 GB/s	4.3×	172 ms
Local NVMe	~7 GB/s	2.4×	307 ms
10 GbE or object storage	~1.25 GB/s	0.43×	1.7 s

Only the last row loses, and it loses to a 0.731 s recompute by a factor of 2.3. Every tier a datacenter actually offers, down to and including local flash, is worth using for agent session state. That is a strong claim and it is worth pausing on, because the instinct in a serving team is that KV cache is a GPU-memory thing and everything else is too slow. The arithmetic says the opposite: the cache is a storage problem with an unusually generous budget.

Now the capacity side, which is why any of this matters.

Read the two blocks together and the architecture writes itself: keep the working set in HBM, demote the parked sessions one tier down, and let the break-even decide how far down you may go before recompute becomes the cheaper backstop. That is Mooncake's disaggregated KV store, and its design is less an invention than an acceptance of this division.

The caveat that keeps it honest: bandwidth is not the only currency. §19.6 showed a fleet that was PCIe-bound at 45 sessions, and every byte spent demoting a parked session is a byte not available for swapping a stalled one. The tiers are shared, the two mechanisms compete, and a deployment that sizes them independently will discover the interaction in production.

Instructor notes

Minutes: 8. Board: The tier table, ratios column only — the times are for the notes. Draw the line under NVMe. Ask the room: "Which row surprises you?" Usually NVMe. Then: "what would have to be true for NVMe to lose?" MHA, or a short session, or a much faster prefill — all three are in §20.2.

20.4 What the client's context policy does to the server's cache

The centerpiece, and the one place this course reverses a conclusion it reached earlier.

Lecture 4 costed three assemble policies over a 20-turn session and ranked them by cumulative prompt tokens — the quantity a client pays for and can measure. Recall the ledger exactly: a 2,000-token preamble, 300 tokens appended per turn, 20 turns.

Policy	Cumulative prompt tokens submitted
Keep until full	97,000
Sliding window, pinned, W = 3,000	83,500
Summarize and compact, trigger 3,000, summary 500	75,100 including compaction traffic

Summarize-and-compact won by 23%, and Lecture 4's conclusion was that the three differ by little in tokens and enormously in failure mode. Both halves of that stand. What Lecture 4 could not ask, not having built a prefix cache yet, is a different question: of those submitted tokens, how many must actually be computed?

The answer depends on a property Lecture 13 established and that is easy to state and easy to forget: reuse requires an exact prefix match. The KV for a token depends on every token before it, so a cached prefix is reusable only if the new prompt begins with exactly the same tokens. The moment a prompt differs at position i, everything from i onward must be recomputed — no matter how much of it is textually familiar.

Now walk the three policies with a perfect, unbounded prefix cache.

Put the two columns beside each other.

Policy	Submitted (Lecture 4)	Computed with a perfect prefix cache	Prefill time
Keep until full	97,000 — worst	7,700 — best	0.21 s
Sliding window, W = 3,000	83,500	32,000 — worst	0.87 s
Summarize and compact	75,100 — best	11,200	0.31 s

The prefill column costs weight GEMMs only, at Lecture 13's rate of 2N = 13.48 GFLOP/token ÷ 494.5 TFLOP/s = 27.26 µs/token, so that 0.21 s is exactly the figure Lectures 13 and 18 have been carrying for a perfectly-cached 20-step task. Adding §20.2's quadratic attention term raises all three numbers — these prompts run to 7,700 tokens, not 16,384, so the term is smaller here than the 17.37 µs of the standing assumptions — and it raises them in the same direction, because every prefill term is linear in the number of recomputed tokens. The ranking is set by the token column and the seconds only price it.

The ranking inverts. The policy that submits the most tokens computes the fewest, by 4.2× over the sliding window and 1.45× over compaction. And the sliding window — which looks like the sensible engineering compromise, and which is what most agent frameworks implement by default — is the worst of the three on the server by a wide margin, while looking like a 14% improvement from the client's seat.

The mechanism is worth naming precisely, because it generalizes past these three policies. Pure appending preserves the prefix; any edit to the middle of the context invalidates everything after it. A sliding window edits the middle every single turn, so it destroys the cache every single turn. Compaction edits the middle once, so it pays once — 3,800 tokens of penalty at turn 12 — and then appends cleanly again. Keep-until-full never edits, so it never pays.

Which raises the question of why a rational client would ever do the expensive thing, and the answer is pricing.

Under flat pricing the client is rewarded for destroying the server's cache. Under cached-input pricing — which every major API now offers — the incentives line up, and the client's cheapest policy becomes the server's cheapest policy. The pricing model, not the code, decides whether context management helps or hurts. That is a systems result about an economic mechanism, and it belongs in this course precisely because it is invisible from either side alone.

Two honest limits. First, this assumed an unbounded cache with no eviction; under pressure, keep-until-full's 7,700-token prefix is also the largest thing to hold, and Lecture 13's eviction policy decides whether it survives between turns — §20.5 takes that up. Second, Lecture 4's ranking by failure mode is untouched: keep-until-full still dies abruptly when the window fills, and that remains the reason not to choose it for long sessions. The correct reading is not "keep-until-full wins" but "the token ledger was never measuring the thing that costs money."

Instructor notes

Minutes: 23. Never cut. Board: Lecture 4's three-row table on the left, untouched. Derive the computed column on the right, policy by policy, and draw the crossing arrows between the two "best" cells at the end. Ask the room: Before deriving: "Which policy is kindest to a prefix cache?" Most say compaction, because it submits the fewest tokens. Let them commit. Then derive keep-until-full's 7,700 and watch it land. Ask the room: "Your framework's default is a sliding window. What is it costing you?" 4.2× prefill, invisibly. Expect confusion: Two, reliably. (1) "Why doesn't the window's content hit? It is the same text." Because it sits at a different offset behind a prefix that no longer matches — text identity is not prefix identity. Redraw the two prompts aligned to make it visual. (2) Students think the compaction penalty is the summary call. It is mostly the 3,300-token prefill of the compaction input plus the 500-token break; name both.

20.5 Retention: how long to keep a session that may never return

Lecture 19 answered "keep, swap, or discard" for a stall of known duration. Between turns, the duration is not known, and may be infinite: the user closed the tab. So retention becomes a bet.

Take the ingredients from Lecture 19. Holding a session costs f = 3.44% of the card's KV budget per second. Rebuilding it costs C = 0.731 s of GPU. If the session returns with probability p within the retention window, the expected saving from having kept it is p · C, and the cost of keeping it for T seconds is f · T. Keep while the first exceeds the second:

At p = 1 this is exactly Lecture 19's keep-versus-discard break-even, 21.3 s, which is the check that the two derivations agree. At p = 0.5 — half of sessions never come back — the TTL halves to 10.7 s. At p = 0.1 it is 2.1 s, which is shorter than a single tool call, meaning a fleet of mostly-abandoned sessions should evict aggressively between every turn.

Three observations on that formula, in order of how much they should change your design.

The TTL is not a constant, and treating it as one is the common failure. It scales with the rebuild cost, so a long session deserves a longer TTL than a short one; it scales inversely with occupancy, so it is architecture-dependent; and it scales with the return probability, which is a property of the application, not the engine. A coding agent between steps has p near 1. A customer-support session after the user's last message has p near 0.1. Same engine, 10× different policy.

Swap changes the question rather than answering it. With host DRAM available, a session need not be discarded at TTL expiry — it can be demoted, at 34 ms, and the TTL for the DRAM copy is set by DRAM pressure rather than HBM pressure. So the real structure is a TTL per tier, each with its own f, which is how the storage hierarchy of §20.3 and the retention question of this section are the same mechanism.

Reference counts beat timers when the structure is available. Lecture 18's §18.5 made this point for declared DAGs: if the server knows a prefix has k unscheduled descendants, it pins until k reaches zero instead of guessing. The same applies here — an application that declares "this session is finished" converts a probabilistic TTL into a deterministic free, and the reason engines use timers is not that timers are better but that nobody sends the message.

Instructor notes

Minutes: 8. Board: T* = p · C / f, then the three values of p and the three TTLs. Write "same constant as Lecture 19 at p = 1" beside it. Ask the room: "What is p for the agent you built in Assignment 2?" Near 1 between steps, near 0 after the task completes — so the TTL should differ by an order of magnitude within one session's lifetime, which no fixed timer expresses.

20.6 Migration is cheaper than rebuilding

Routing and load balancing (Oct 26) introduced migration as a load-balancing mechanism and priced it in the abstract. With session state on the table, price it concretely.

So for agent sessions, the Llumnix-style answer is clearly right, and more strongly than it is for chat: the longer the session, the better migration looks, because transfer is linear in S while rebuild is superlinear. The same reason NVMe stayed in play in §20.3.

The complication is the one Lecture 19 raised and this lecture cannot dissolve: the session is not only its KV. It is also a sandbox with a filesystem and possibly a live interpreter, and a 43 ms KV migration paired with a 2 s container rebuild on the destination is a 2 s migration. The migratable unit is the slowest half of the session, which is an argument either for stateless sandboxes or for DeltaBox-style checkpointing, and it is not a KV-cache decision at all.

Instructor notes

Minutes: 7. Board: 43 ms against 0.731 s, then the same pair one length down: a 4,096-token session is 0.537 GB, so 0.537 ÷ 50 = 10.7 ms to move against 4,096 × 31.60 µs = 129 ms to rebuild — 12× at 4K against 17× at 16K. Write both ratios and circle the direction. Ask the room: "Which way does that ratio go as the session grows, and why?" Transfer is linear in S, rebuild is superlinear, so migration wins by more the longer the session — the same asymmetry that kept NVMe in play in §20.3. It is the third appearance of one fact today; say so. Ask the room: "You have a 43 ms KV migration and a 2 s container to rebuild. What did you just migrate?" Nothing. The answer is the section. Expect confusion: Students price migration against the stall rather than against the rebuild. Migration competes with recompute-on-arrival, not with staying put; staying put is free. If short on time: state the 17× and hand the 4K pair to the room as arithmetic.

20.7 Memory above the cache

(Reading section — not scheduled in class.)

Everything so far is the KV cache: a derived, exactly-reconstructible artifact. Agents also have memory in the ordinary sense — retrieved documents, scratchpad notes, summaries of previous sessions, a vector index — and it is worth being precise about how differently the two behave, because the vocabulary invites conflating them.

	KV cache	Agent memory
Derived from	The exact token sequence	Whatever the application decided to write
A miss causes	Recomputation, then identical output	A different answer
Correctness if lost	Unaffected	Unbounded degradation
Invalidation	Automatic: change a token, the suffix is invalid	Manual, and usually not implemented
Who decides retention	The engine	The application

The fourth row is where most production incidents live. A KV cache cannot serve a stale answer, because the key is the content. An agent memory store absolutely can — a summary written three turns before the plan changed is retrieved with full confidence and no signal that it is obsolete.

Which is the sharper reading of MemGPT that Lecture 3 asked you to hold. Its analogy to virtual memory is exact everywhere except the one place that matters: there is no page fault. A paging system detects the miss and stalls the instruction; an agent that has evicted a constraint from its context does not know it is missing and produces a fluent answer that violates it. Lecture 4 called a context miss silent; here is the storage-side statement of the same fact. Every mechanism in §20.2 through §20.6 is safe precisely because it operates below that line, on the derived artifact where a miss costs time and nothing else.

20.8 The invariant that makes all of this safe

One sentence, and it licenses the entire lecture.

That is why a serving team can be aggressive here in a way it can be nowhere else in the stack. The policy space of §20.3 through §20.6 is large, the penalties are computable in advance, and the worst outcome of a bad choice is a slow session. Contrast §20.7, one layer up, where the same aggression changes what the model says.

It also tells you what to durably store and what never to: the transcript, always — Lecture 4 made recovery equal to replay — and the KV cache, never, because at 2.94 GB/s of break-even bandwidth, recomputing it is cheaper than most ways of having saved it.

Instructor notes

Minutes: 5, and they are the close of this note rather than of the meeting — Lecture 21 follows in the same room. Board: One line: "the transcript is the state; the cache is a function of it." Under it, two columns headed costs time and costs correctness, and let the room sort today's six mechanisms — demote, compress, evict, migrate, discard, compact — into them. Compaction is the only one that crosses, which is the handoff to §20.7. Ask the room: "Name the one thing in today's lecture that can change an answer." Compaction, because the summary is written by a model and then becomes input. Everything else is derived. Expect confusion: Students hear "the cache can never be wrong" as "the cache can never hurt." It can, expensively — §20.4's 4.2× is entirely a cost, and costs are what this note is about. If short on time: the one board line, then point at Lecture 21.

Discussion seeds

1. The default nobody chose. Most agent frameworks ship a sliding window, which §20.4 makes the worst of the three policies on the server by 4.2×. What is the smallest change to a framework's assemble function that recovers most of the gap, and what does it give up in the context-window-overflow failure that Lecture 4 said was keep-until-full's real defect?
2. Who pays for the cache. Exercise 4 puts the client's indifference point at a 36% cached-input discount. Should a platform publish computed tokens on the invoice alongside submitted tokens — and could a client verify that number if it wanted to?
3. Where p comes from. §20.5's TTL is p · C / f, and p is the one term the engine is never told. Which of Lecture 19's free structural signals — the tool name, the interception type, the declared DAG — would you use to estimate it, and what does a wrong estimate cost in each direction: too high, too low?
4. One bus, two mechanisms. Exercise 5 shows swapping stalled sessions and demoting parked ones competing for the same 64 GB/s. Would you give them separate reservations or one queue with priorities, and which quantity in §20.5 tells you the priority?
5. When to break the invariant. §20.8 says never persist KV. Construct a workload where persisting it is right anyway — a very long fixed preamble, a cold fleet, a low arrival rate — and find the break-even between storage cost and recompute cost. What property of your workload makes it the exception?
6. The line compaction crosses. A compaction summary is written by the model and then becomes authoritative input, so it moves bytes from the derived side of §20.7 to the authoritative side. What would you have to log to make a compaction reversible, and is a reversible compaction still worth doing?

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
Break-even bandwidth, GQA-8	4.15 GB/s at 4K, 2.94 GB/s at 16K	131,072 B ÷ prefill µs per token
Break-even bandwidth, MHA	16.6 GB/s at 4K, 11.7 GB/s at 16K	524,288 B ÷ same
Tiers that lose to recompute	Only 10 GbE and object storage (0.43×)	§20.3 table
Fleet state, 10,000 sessions	21.5 TB = 345 H100s of HBM, or 22 DRAM nodes	10,000 × 2.147 GB
Computed tokens, keep-until-full	7,700 (0.21 s) from 97,000 submitted	Pure append; 12.6×
Computed tokens, sliding window	32,000 (0.87 s) from 83,500 submitted	Window slides, prefix breaks each turn
Computed tokens, summarize-compact	11,200 (0.31 s) from 75,100 submitted	One invalidation at turn 12
Cost under cached-input pricing	$0.050 / $0.112 / $0.053	$3 uncached, $0.30 cached per Mtok
Retention TTL	p · C / f = 21.3 s at p = 1, 2.1 s at p = 0.1	§20.5
Migrate versus rebuild	43 ms versus 0.731 s = 17×	2.147 GB ÷ 50 GB/s

Self-check

Exercises

1. Tier a real deployment. A 70B model at GQA-8 with 320 KiB per token and 140 µs per token of prefill at 8,192 tokens of context. (a) Break-even bandwidth. (b) Which rows of §20.3's table now lose? (c) How does the answer change if the cache is compressed 2× with a CacheGen-style encoding? Solution sketch: (a) 327,680 B ÷ 140 µs = 2.34 GB/s — lower than the 7B's 2.94, because the bigger model is more expensive to rebuild per byte stored. (b) The same single row, and no more: local NVMe at ~7 GB/s now clears the bar by 3.0× instead of 2.4×, while 10 GbE and object storage at ~1.25 GB/s still lose, now by 1.9× instead of 2.3×. A bigger model narrows every gap in the loser's favour without flipping a row — which is the honest version of the claim, and worth noticing: the tier table is stable under model size. (c) Halving the bytes halves the required bandwidth to 1.17 GB/s, which finally puts 10 GbE in play at 1.07× — compression buys tiers, which is exactly the argument for spending quality on it, and note how little margin the purchase comes with.
2. The inversion, on a different session shape. Rerun §20.4 for a 40-turn session with the same 2,000-token preamble and 300 tokens per turn, with the window still at W = 3,000 and compaction still triggering at 3,000 with a 500-token summary. (a) Computed tokens under each policy. (b) Does the ranking hold? Solution sketch: Keep-until-full computes the final length: 2,000 + 39 × 300 = 13,700. The window is pure-extension through turn 11 (5,000) then recomputes 3,000 for turns 12–40: 5,000 + 29 × 3,000 = 92,000. Compaction now fires repeatedly: the first firing at turn 11, and each subsequent one once the 500-token summary has re-accumulated to 3,000 at 300 tokens per turn, which takes ⌈2,500/300⌉ = 9 turns — so four firings in 40 turns, at turns 11, 20, 29, and 38. Each costs a 3,300-token compaction prefill plus a 500-token restart = 3,800. Of turns 12–40, four are those restarts and the other 25 are clean extensions at 300: computed = 5,000 + 4 × 3,800 + 25 × 300 = 27,700. The ranking holds and widens: the window is 6.7× keep-until-full where at 20 turns it was 4.2×, and compaction has slipped from 1.45× to 2.0×. Repeated invalidation compounds — its cost is linear in the number of firings, which is linear in session length — whereas appending never pays at all.
3. Retention under abandonment. A fleet where 30% of sessions never return after any given turn, and the mean inter-turn gap for those that do is 4 s. (a) TTL from §20.5. (b) What fraction of returning sessions find their cache gone? (c) What does that cost per 1,000 turns? Solution sketch: (a) p = 0.7 → T* = 0.7 × 21.3 = 14.9 s. (b) With a 4 s mean gap and a 14.9 s TTL, almost none — under an exponential gap distribution, exp(−14.9/4) ≈ 2.4%. (c) 1,000 turns × 0.7 returning × 0.024 × 0.731 s ≈ 12.3 s of GPU per 1,000 turns, which is negligible — and that is the finding: at these numbers the TTL is generous enough that misses are not the problem. The problem is the 30% that never return holding capacity for 14.9 s each.
4. Pricing as a control knob. You run the serving platform and set the prices. Given §20.4's computed-token figures, find the cached-input discount at which a rational client is indifferent between keep-until-full and the sliding window, and say what that implies. Solution sketch: Let the cached rate be c per Mtok against $3 uncached. Keep-until-full costs 7,700 × 3 + 89,300c; the window costs 32,000 × 3 + 51,500c. Setting them equal: 23,100 + 89,300c = 96,000 + 51,500c → 37,800c = 72,900 → c = $1.93, a 36% discount. Any discount deeper than that makes keep-until-full strictly cheaper for the client. The illustrative $0.30 rate used in §20.4 is a 90% discount, far past the threshold, so under that pricing the incentives are already aligned — but a platform charging a flat rate is paying its clients to destroy its own cache, and the threshold says how little of a discount it would take to stop.
5. The competing-bus problem. Using §19.6's swap-bound fleet (45 sessions, 1.431 GB/s each) and a policy that also demotes parked sessions to host DRAM, find the maximum fraction of sessions that can be tiered before PCIe saturates, and propose a rule for choosing which. Solution sketch: Swap already consumes 45 × 1.431 = 64.4 GB/s, which is the whole bus — so at that operating point, zero. Demotion and swap are the same traffic in different clothes, and the fleet must choose. A defensible rule: demote by expected time-to-return, longest first, since a session that will not return for a minute pays the transfer once and frees capacity for many stalls, whereas a session returning in 1.5 s pays the same transfer for a fraction of the benefit — which is §20.5's TTL used as a sort key rather than a threshold.

Reading guide

Required — Continuum. The paper that treats retention and scheduling as one mechanism. Read the motivation first and check it against Lecture 19: its claim about engines evicting a finished request's cache while the session is about to return is exactly Lecture 19's keep-versus-discard question, arrived at from the scheduler's side. Then read the TTL design for what it takes as input, and hold §20.5's T* = p · C / f beside it — where does the paper get its p, and does it vary per session or per deployment? Finally, read the program-level first-come-first-served component carefully, because it is Lecture 22's subject, which the Nov 18 meeting already covered: the paper's position is that retention without a matching admission order does not work, and you should be able to say why before class.

Optional — Mooncake. Third appearance in this course, and this time read only the storage architecture: the KV store, the transfer engine, and how it decides what lives where. Ask whether its tiering matches §20.3's arithmetic or predates it.

Optional — CacheGen. Read for the encoder and the quality-versus-bandwidth curve. The interesting question is not whether it works but where on §20.3's table it moves you, and whether the tier it unlocks was worth the quality it spent.

Optional — prediction-based KV-cache management. §20.5 with learning in place of a constant. Read the feature set — what does it use to predict return time? — and compare against the free structural signal Lecture 19 found in the tool name.

Optional — KV-cache management survey and MemGPT. The survey is a map; use it to place every KV paper this course has assigned and to find the region none of them covered. MemGPT is the §20.7 re-read: same analogy, different layer, and the missing page fault is the whole difference.

Looking ahead

Lecture 21 — later in this same meeting: "Efficient agent serving systems II (continued): multi-agent workloads and performance optimization." Today's single session becomes k of them sharing a parent: the fan-out that Lecture 18 scheduled without a barrier gets its barrier back, and the join makes the slowest worker — not the average one — the latency. It is also where the course turns reflexive and asks whether spawning a fleet of agents to optimize a system is worth what it costs, which is a question your Assignment 5 write-up answers empirically — due Dec 2, and presented the week before at the student sharing sessions on Nov 30 and Dec 2.

Lecture 22 — already covered, Wednesday, November 18, where it shared the meeting with Lecture 18: "Efficient agent serving systems I (continued): scheduling agentic programs." Continuum's insistence that retention and admission order are one mechanism was that subject in full, with Autellix scheduling programs whose length nobody knows and Teola optimizing the graph inside a single request.

Where the relationship inverts — the models stop being the thing served and become the thing doing the work — is optional content on this calendar rather than a lectured block; Lecture 24's notes and the readings are up if you want it. Two dates remain. Assignment 5 (optimize the full stack) is due Dec 2, 11:59pm, and the final project report is due Dec 8. §20.4 is unusually direct ammunition for any project that touches prefix reuse: traces are produced by clients with context policies, and recognizing which policy produced a trace tells you what its reuse structure can possibly be.

One thing to carry out of the room. Every mechanism in this note was licensed by one sentence — the cache is a pure function of the transcript, so losing it costs time and never correctness. That is why the policy space here is safe to be aggressive in, and it is also why the largest number in the note, §20.4's 4.2×, was not produced by a serving decision at all. It was produced by a client choosing a context policy for reasons that had nothing to do with the machine.