SoloLLM started out as a challenge to see if I could train a base model from scratch on consumer hardware that could reach GPT-2-level performance. After a few model iterations, it had already reached GPT-2 small level performance. At that point the project shifted to a more open small model research project.
The refined goal for this project is to use SoloLLM as a research platform for understanding how to build stronger small language models from scratch with hardware constraints
| Model | Params | Tokens | Status | Hugging Face |
|---|---|---|---|---|
SoloLLM-1.0-152M-Base |
151.9M | 9.8B | Strongest completed SoloLLM 1.0 checkpoint | model |
SoloLLM-1.0-123M-Base |
123.6M | 9.8B | Smaller-than-GPT-2 comparison checkpoint | model |
SoloLLM-1.0-152M-Baseis the strongest checkpoint so far. It beats GPT-2 small across the fixed base-model evaluation suite, but it is slightly larger than GPT-2 small.SoloLLM-1.0-123M-Baseis slightly smaller than GPT-2 small. It wins the normalized multiple-choice suite and several external checks, but it does not beat GPT-2 small across every metric.- Data mix seemed to have the largest impact on model performance. A reasoning-heavy mix improved some multiple-choice and repetition metrics, but hurt broad language-modeling perplexity. A broader blend with dropout
0.0produced the best overall result.
Benjamin Maxwell