Design an AI code completion system like GitHub Copilot's inline suggestions.

30-second answerSay your answer out loud first, then reveal.

An IDE extension gathers context on each pause in typing (code before and after the cursor, the open file, relevant snippets from other open or recently edited files, imports, language), builds a fill-in-the-middle (FIM) prompt within a token budget, and sends it to a low-latency completion service running a small, fast code model. Debounce keystrokes, cancel stale requests, cache, and post-process (truncate at sensible boundaries, filter low-quality or insecure suggestions). The north-star metrics are acceptance rate and retained code at about 300ms latency.

Code completion flow: a typing pause triggers context gathering and a FIM prompt builder; on a local cache miss the request goes to a nearest-region completion service and small code model, then post-processing produces ghost text and telemetry, while a new keystroke cancels the in-flight request.

Key design points

Latency is everything: suggestions must appear within a few hundred ms or they're useless. Use small models (1–15B class) optimised for speed, nearest-region serving, persistent connections, speculative decoding, and short max output (a line or block).
Fill-in-the-middle: models trained with <prefix> ... <suffix> ... <middle> formats use code after the cursor, which is crucial for good insertions.
Context retrieval: the best gains often come from relevant snippets of other files (Jaccard or embedding similarity over recently viewed files, symbol definitions via the language server).
Request economics: many requests per user per hour. Debouncing, caching (typing through a suggestion should reuse it) and cancellation cut load significantly.
Post-processing: cut at syntactically sensible points (using indentation and brackets); avoid repeating existing code; filter secrets; optionally block suggestions matching public code (licence concerns).
Evaluation: offline (exact match / execution-based on held-out repositories, FIM benchmarks); online (acceptance rate, characters retained, latency percentiles, by language).
Privacy: enterprise settings for no retention or training; on-prem options.

Separate from chat/agent features: the multi-file "agent mode" uses larger models and different latency expectations (see the Agentic AI guide).

PreviousDesign a ChatGPT-like consumer chat application for 10 million daily active users.

You understood something today that you didn't yesterday.

Design an AI code completion system like GitHub Copilot's inline suggestions.

Key design points

Related