Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

1-bit LLMs (BitNet b1.58)

BitNet b1.58 is a Transformer language model whose weights are all ternary, each one -1, 0 or +1, which takes log2(3) ≈ 1.58 bits per weight and turns most multiplications into additions.

Last updated: 08 Oct, 2026 · NumPy

The BitNet b1.58 paper (Ma et al., February 2024) reports a model with ternary weights that matches a 16-bit Transformer of the same size on perplexity and on end tasks, at lower memory and latency. The first BitNet had two values per weight, -1 and +1; b1.58 adds zero, which is why the name says 1.58 bits and not 1.

Ternary weights and the 1.58 bits

What a ternary weight is · from the Fine-Tuning LLM Models Crash Course video · 1:24:48 to 1:25:55

Every weight in the model is one of three numbers, not a 32-bit or 16-bit float. Three values need log2(3) bits to tell apart, which is where 1.58 comes from.

ExampleBits per ternary weight and the memory of a 7B model
import numpy as np

print("bits to tell 3 values apart:", round(float(np.log2(3)), 4))
print("7B weights at 1.58 bits:", round(7e9 * np.log2(3) / 8 / 1e9, 2), "GB   (FP16: 14.0 GB)")

The number is 1.585 bits. At that width 7 billion weights occupy 1.39 GB, against 14.0 GB at FP16. Real implementations pack the values into whole bits, so the stored size is a little higher; the figure is the information content.

Adding and subtracting instead of multiplying

Why ternary weights make the matrix multiply cheap · from the Fine-Tuning LLM Models Crash Course video · 1:32:22 to 1:35:28

In a 16-bit model most of the cost is multiplying and adding floats. With weights in {-1, 0, +1}, a product w·x is x, -x or 0, so a dot product becomes: add the inputs where the weight is +1, subtract them where it is -1, skip them where it is 0. The zeros also switch inputs off, which the paper calls feature filtering.

ExampleA ternary matrix times a vector, written as adds and subtracts
import numpy as np

rng = np.random.default_rng(7)
W = rng.normal(0, 0.5, (4, 6))
x = rng.normal(0, 1, 6)
gamma = np.abs(W).mean()
Wt = np.clip(np.round(W / (gamma + 1e-8)), -1, 1)

# with weights in {-1, 0, +1} a dot product is: add where +1, subtract where -1, skip where 0
out = np.array([x[row == 1].sum() - x[row == -1].sum() for row in Wt])
print("add/subtract only:", out.round(4))
print("Wt @ x           :", (Wt @ x).round(4))
print("gamma * out      :", (gamma * out).round(4))
print("W @ x (original) :", (W @ x).round(4))

The first two rows of the output are identical: the add and subtract loop and Wt @ x give the same numbers, 0.4739, 2.9333, -0.4166 and 2.5954, with no multiplication in the first. Multiplying by γ restores the scale of the original matrix; here that product is not close to W @ x, because this ternary matrix is a coarse copy of a random matrix. That is why the model is trained with ternary weights, covered below.

Absmean quantization

BitNet maps a float matrix W to ternary values with the absmean function. γ is the average absolute value of the matrix, W is divided by it, and RoundClip rounds each entry to the nearest integer and clamps it to the range -1 to +1:

Absmean quantization; ε is a small constant that avoids dividing by zero
A 4 by 6 float weight matrix is divided by its average absolute value 0.3471, rounded and clipped to the three values minus one, zero and plus one.
ExampleAbsmean quantization of a 4 x 6 matrix
import numpy as np

rng = np.random.default_rng(7)
W = rng.normal(0, 0.5, (4, 6)).round(2)               # a small weight matrix
gamma = np.abs(W).mean()                              # absmean: average absolute value
Wt = np.clip(np.round(W / (gamma + 1e-8)), -1, 1)     # RoundClip to {-1, 0, +1}
print("W =\n", W)
print("gamma =", round(float(gamma), 4))
print("ternary W =\n", Wt.astype(int))
print("values used:", sorted(set(Wt.astype(int).ravel().tolist())))

γ is 0.3471. An entry with |W| below about half of γ rounds to 0, an entry between half and 1.5 times γ becomes ±1, and anything larger is clipped to ±1. The result uses exactly the three values -1, 0 and 1.

Training with ternary weights from scratch

BitNet b1.58 replaces each nn.Linear layer of the Transformer with a BitLinear layer. The model is trained from scratch with 1.58-bit weights and 8-bit activations: the ternary weights are used in the forward pass during training, so the network learns values that still work after rounding. Rounding a finished 16-bit model to ternary afterwards is a different thing and is not how BitNet is made. The next block shows the damage on a random matrix.

ExampleTernarizing a matrix that was never trained for it
import numpy as np

rng = np.random.default_rng(3)
W = rng.normal(0, 0.02, (256, 256))                  # a "trained" matrix: no ternary training behind it
X = rng.normal(0, 1, (256, 100))
Wt = np.clip(np.round(W / np.abs(W).mean()), -1, 1)
approx = np.abs(W).mean() * (Wt @ X)
exact = W @ X
rel = np.linalg.norm(approx - exact) / np.linalg.norm(exact)
cos = (approx * exact).sum() / (np.linalg.norm(approx) * np.linalg.norm(exact))
print("share of zeros in ternary W:", round(float((Wt == 0).mean()), 3))
print("relative output error:", round(float(rel), 3), " cosine:", round(float(cos), 3))

31% of the entries become zero. The ternary output has a cosine similarity of 0.888 with the exact output and a relative error of 0.511, so it points the same way and is still a rough copy. Training with the rounding in place is what lets the paper's model close that gap.

BitNet b1.58 against a 16-bit LLaMA at 3 billion parameters

The paper compares BitNet b1.58 with a 16-bit LLaMA-style model of the same size and training data. At 3 billion parameters (Table 1 of the paper):

Model, 3BMemoryLatencyPerplexity
LLaMA (16-bit)7.89 GB5.07 ms10.04
BitNet b1.582.22 GB1.87 ms9.91

The paper reports that BitNet b1.58 starts to match the full-precision model at 3 billion parameters. A later release, BitNet b1.58 2B4T (2025), is a 2-billion-parameter model trained on 4 trillion tokens with open weights and inference code for GPU and CPU. Standard GPUs have no ternary kernel, so the latency and energy gains need software and hardware written for these weights; the paper itself calls for such hardware.

1-bit LLMs vs 4-bit quantization

FP16 model4-bit PTQ modelBitNet b1.58
Bits per weight16About 41.58
When the weights are roundedNeverAfter trainingDuring training, from scratch
Multiplications in the weightsFloat multipliesDequantized, then float multipliesAdds and subtracts
Existing models can use itYesYesNo, a new model must be trained

Where you use 1-bit LLMs

  • Research and edge targets. A model that needs 1.39 GB per 7 billion weights could run where 14.0 GB cannot.
  • Reading the quantization trend. 8-bit, 4-bit and ternary are points on one curve of bits against quality; see Quantization.
  • Hardware design. Adds and subtracts cost less energy than float multiplies, the case for ternary-aware chips.
Watch out. The 1.58-bit saving applies to models trained as ternary. A 16-bit model you already have cannot be turned into BitNet by rounding its weights; for that model use 8-bit or 4-bit quantization.
Try it yourself
  • In the add and subtract block, change the seed from 7 to 8 and check that the two lines that must match still match.
  • In the absmean block, multiply W by 10 before computing γ and compare the ternary matrix with the first one.
  • In the last block, change the 0.02 to 0.2 and read how the zero share and the cosine move.
PreviousQuantization

Every expert started right here.