Why do LLMs struggle with tasks like counting letters ("how many r's in strawberry"), arithmetic, or reversing facts?

30-second answerSay your answer out loud first, then reveal.

Many such failures trace back to how models see text and how they learn. Tokenization hides characters (the model sees chunks like "straw" and "berry", not letters) and splits numbers inconsistently. Autoregressive models compute a fixed amount per token, so long multi-step arithmetic done "in their head" is error-prone. Training on one-directional statements doesn't automatically teach the reverse ("A is B" doesn't give "B is A": the reversal curse). The fixes are tools (code execution), reasoning steps, and better tokenization.

1. Character-level tasks

"strawberry" may tokenize as ["str", "aw", "berry"], so letters aren't directly visible as units.
The model must have memorised the spelling of each token. Workarounds: ask it to spell the word out letter by letter first, or use code.

2. Arithmetic

Numbers are tokenized into chunks inconsistently (e.g. "12345" → ["123", "45"]), which misaligns place values.
Multi-digit multiplication needs many sequential steps. Without writing intermediate steps (CoT), the fixed per-token computation isn't enough.
Mitigations: chain-of-thought, digit-level tokenization (some models split numbers into individual digits), or a calculator / code tool, which is the best option in production.

3. Reversal curse (Berglund et al. 2023)

Trained on "X's mother is Y", models often fail "Who is Y's son?" Learned associations are directional.
Relevant for knowledge retrieval: an entity may be "known" from only one direction.

4. Other characteristic weaknesses: precise counting over long lists; exact recall of rare facts; consistency across long outputs; planning tasks requiring search.

Interview framing. "These aren't random bugs. They follow from tokenization, autoregressive computation, and statistical learning. In production I route such sub-tasks to deterministic tools."

PreviousHow do you extend a model's context window (e.g. from 8K to 128K)? Explain RoPE scaling methods.

Little by little, you're building something great.

Why do LLMs struggle with tasks like counting letters ("how many r's in strawberry"), arithmetic, or reversing facts?

1. Character-level tasks

2. Arithmetic

3. Reversal curse (Berglund et al. 2023)

Related