Links & Resources
Microsoft's research team shipped bitnet.cpp, the official inference framework for 1-bit LLMs. It does something that was supposed to require a $10,000 GPU: it runs a 100B-parameter BitNet model on a single CPU at 5-7 tokens per second - human reading speed. On x86 it cuts energy use by up to 82.2% and runs up to 6.17x faster than full-precision inference. 39.3K GitHub stars. MIT licensed. Free.
The 1.58-Bit Trick
Normal LLMs store each weight as a 16-bit float. BitNet models store each weight as one of three values: -1, 0, or +1. That's 1.58 bits per weight instead of 16. Expensive floating-point multiplication collapses into simple addition and subtraction - which any CPU handles natively and cheaply.
That single change kills the GPU requirement. bitnet.cpp is the optimized kernel layer that makes it fast and lossless on the CPU already in your laptop.
The Numbers
| CPU | Speedup vs. full precision | Energy reduction |
|---|---|---|
| ARM | 1.37x - 5.07x | 55.4% - 70.0% |
| x86 | 2.37x - 6.17x | 71.9% - 82.2% |
Larger models see bigger gains. The headline: a 100B BitNet model on one CPU at 5-7 tokens/sec.
Honest caveat: 100B is the demonstrated ceiling. The model Microsoft actually ships is BitNet-b1.58-2B-4T (2.4B params, 4T tokens) - that's the one you'll download below.
Quick Start
Runs on Linux, Windows, macOS. Needs Python 3.9+, CMake 3.22+, Clang 18+.
git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet
conda create -n bitnet-cpp python=3.9 && conda activate bitnet-cpp
pip install -r requirements.txt
Download the model and build:
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
Chat with it:
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv
That's it. Local LLM on CPU. No GPU, no cloud bill, no per-token fee.
Why This Matters
Local inference has been gated behind expensive GPUs, which pushed most people to cloud APIs and a monthly bill. BitNet changes the unit economics: if a capable model runs at reading speed on a CPU you already own, the marginal cost of inference drops toward zero - private, offline, free.
It won't replace GPUs for training or frontier serving. But for running a solid local assistant on your own hardware, it removes the hardware excuse entirely.