Memory Systems for LLM Applications

"Memory" is one of the most conflated terms in AI engineering interviews, and the conflation is the point of the question. A candidate who says "we'll add memory so it remembers the user" without saying which memory — the live conversation, a fact stored between sessions, a record of a past failed attempt, or a scratchpad the model writes to itself — has not actually answered anything. The interviewer is testing whether you can decompose "memory" into distinct systems with different lifetimes, different storage, different retrieval paths and different failure modes, because in production those four things are built, tuned and debugged separately even though a demo can fake all four with a single growing prompt.

The core tension underneath every memory design is the same one that shows up in RAG and in context management generally: a language model has no persistent state of its own between calls. Everything it "knows" about the current interaction is whatever text is in the prompt at that call. Memory systems are entirely about what text you put in the prompt, from where, and when — nothing is stored "in the model." That reframing is useful in an interview: instead of describing memory as a mysterious capability, describe it as an engineering decision about which facts get materialized into which prompt, sourced from which store, refreshed on which trigger.

This subject builds a taxonomy — short-term, long-term, episodic, working — then works through the two decisions that actually matter in a design: what is durable enough to persist, and whether it should be loaded eagerly or retrieved on demand. It closes with the ways memory goes wrong in production and a concrete walkthrough of how claude-md-and-memory implements this exact taxonomy in a shipping product. Two siblings are worth knowing alongside this one: agent-architectures-and-the-agentic-loop describes the loop that actually reads and writes memory at each step, and rag-architecture-end-to-end covers the retrieval mechanics — chunking, indexing, similarity search — that long-term and episodic memory reuse wholesale.