Design conversation storage and memory for a chat product with millions of users.

Data model (simplified)

conversations(conv_id, user_id, title, created_at, updated_at, settings)
messages(conv_id, msg_id, role, content, tool_calls, model, tokens_in,
         tokens_out, created_at, parent_msg_id /* for edits/branches */)
summaries(conv_id, up_to_msg_id, summary_text)
user_memories(user_id, memory_id, text, embedding, source_conv, updated_at)

Context assembly per turn

1. System prompt + user memories relevant to the current message (semantic retrieval, top-k).
2. Rolling summary of older turns (updated asynchronously when the history exceeds N tokens).
3. Last K messages verbatim.
4. Tool results / retrieved docs for this turn.

Scale considerations

• Write-heavy: two or more messages per turn. Partition by user_id or conv_id.
• Read pattern: recent conversations list + the full thread on open. Cache active threads in Redis.
• Branching / edits: messages form a tree (parent_msg_id) when users edit earlier prompts.
• Large attachments live in object storage; store references.

Privacy and compliance: user-controlled deletion (conversation and memory), retention policies, encryption at rest, export, and a clear setting for whether data is used for improvement.

Memory quality: extraction in the background, deduplication and conflict resolution, a user-visible memory list with edit and delete.

Related

• Previous: Q28. How do you implement token streaming for a chat application end to end?
• Next: Q30. Design an AI writing assistant embedded in a document editor (like a "copilot" in Google Docs or Notion).
• LangGraph