ai engineering / context window / context compaction / ai agents
AI Agent Context Windows: Compaction Without Losing the Plot
A growing agent conversation makes every request more expensive and can slow the system down. In part 2 of the AI systems design interview series, Austin explains context compaction: summarizing conversation state, checking what the summary forgets, and searching earlier logs when details are needed.
He also covers keeping large tool responses outside the conversation, then introduces the persistent memory problem covered in part 3.
Transcript
0:00Here's Part 2 of the AI Systems Design Round that gives you interviews at companies like Meta and
0:05D E Shaw for the AI Engineering position In Part 1 I talk about the agentic loop where
0:11you respond with a thought and an action The action executes some sort of tool call to get
0:16some sort of response which is appended to the conversation log Now when you're appending things to the
0:22conversation log at some point you're going to have an extremely high context window And when the context
0:29window gets very high then things get really expensive things start to slow down and it starts to
0:36have worse performance Which is why next you want to implement compaction And what compaction is is some
0:45sort of rules that can be simple like the context window reached 192 000 Some sort of rule
0:51that says we're going to summarize the conversation state And by doing this you keep costs a lot
0:58cheaper you keep things faster but you have to be careful because it loses some of the memory
1:06it loses some things that is already done So you have to be careful that the compaction quality
1:11is good or the agent has the ability to search past conversation logs when they need to You'll
1:19also want to talk about keeping heavy responses heavy tool call responses out of the conversation state and
1:25having the agent the ability to look through the response instead This is a great strategy for keeping
1:30the context window minimum Now compaction is very important but let's say you want this agent to run
1:38all the time Let's say you're building an AI agent that understands how to triage on call issues
1:44and you already have this giant run log of previous times you've dealt with these issues This is
1:50where you'll have to implement things like memory so it makes things much faster It doesn't have to
1:56learn everything from scratch every time while your entire system's code base is on fire but you're going
2:01to get that part in part three
Join the conversation
Loading conversation…