TensorForge

A deep learning engine built from scratch in C++17 - no PyTorch, no TensorFlow, no external math libraries. Implements tensor operations, a reverse-mode autograd engine, and a training pipeline capable of training a neural network on MNIST from raw bytes to trained weights.

Why this exists

Most ML work today happens on top of frameworks that hide what's actually going on underneath - memory layout, gradient computation, kernel-level performance. This project rebuilds that stack from the ground up: manual memory management, hand-written SIMD kernels, and a computation graph with backpropagation implemented and verified from first principles.

Features

• Tensor: custom N-dimensional array with manual memory management (RAII, deep-copy semantics, move semantics)
• SIMD-optimized matmul: AVX2 + FMA intrinsics with cache-blocked tiling, benchmarked against a naive implementation
• Reverse-mode autograd: dynamic computation graph with topological-sort-based backpropagation, verified against numerical gradient checking
• Neural network layers: Dense (fully connected), ReLU, Softmax + Cross-Entropy loss with a numerically stable implementation
• SGD optimizer and a full training loop with shuffled mini-batches
• MNIST loader: parses the raw IDX file format directly, no external dependencies

Benchmarks

Naive vs SIMD (AVX2 + FMA, cache-blocked) matrix multiplication, measured on an Intel i7 (2019 MacBook Pro):

Size	Naive (ms)	SIMD (ms)	Speedup
64	1.55	0.077	20.1x
128	12.54	0.46	27.5x
256	119.05	3.61	33.0x
512	1418.26	26.05	54.4x
1024	10504.9	218.68	48.0x

Speedup peaks around 512x512 where the cache-blocked access pattern is most effective; it tapers slightly at 1024 due to cache pressure at that working set size.

Results

Training a 2-layer MLP (784 -> 128 -> 10) on MNIST with plain SGD:

• 3 epochs, batch size 64, learning rate 0.1
• Test accuracy: 95.76%, reaching 93.5% after just epoch 1 (see training log below) (fill in your actual final number from training output)
• Gradient correctness verified via numerical gradient checking

Full training log (3 epochs, batch size 64, learning rate 0.1, total time 21.1s on i7-2019):

Epoch 1 done. avg_loss = 0.372369, test_accuracy = 93.5%
Epoch 2 done. avg_loss = 0.201527, test_accuracy = 95.18%
Epoch 3 done. avg_loss = 0.152167, test_accuracy = 95.76%

Architecture

Input (784,)
   -> Dense(784, 128) -> ReLU
   -> Dense(128, 10)
   -> Softmax + Cross-Entropy Loss

Every arrow above is a node in a dynamically built computation graph. Calling backward() on the loss walks the graph in reverse topological order, accumulating gradients into each parameter via hand-written backward functions for matmul, bias-add, and ReLU.

Project structure

include/tensorforge/   public headers (tensor, ops, autograd, layers, loss, optimizer, mnist)
src/                   implementations
benchmarks/            standalone SIMD vs naive benchmark
scripts/               MNIST download helper

Setup

./scripts/download_mnist.sh
mkdir build && cd build
cmake ..
make
./train_mnist       # trains the MLP on MNIST and prints per-epoch accuracy
./bench_matmul      # runs the naive vs SIMD benchmark suite

Requires a C++17 compiler and CMake 3.15+. Uses AVX2/FMA intrinsics, so it targets x86_64 (Intel) machines; porting to ARM/Apple Silicon would mean swapping the immintrin.h calls for NEON equivalents.

What this doesn't do

This is a from-scratch educational/systems project, not a production ML framework. It does not support GPU execution, convolutional layers, or dynamic batching. The scope is intentionally narrow: get every layer of a real training pipeline - tensors, autograd, kernels, optimization - working correctly and fast, without hiding behind an existing library.