LSTM Optimization Algorithm Research

How do SGD, Adam and RMSProp affect the accuracy and convergence rate of an LSTM for tweet sentiment analysis?

My IB Computer Science Extended Essay (2024), awarded an A. Six bidirectional LSTM models were trained from scratch on two tweet datasets, varying only the optimiser and its learning rate, and compared on weighted F1 score and the number of epochs to convergence. The full essay is in extended-essay.pdf (also on Google Drive) and the training code in notebooks/.

Setup

Architecture	TextVectorization encoder (10k vocabulary) → 64-d embedding with masking → bidirectional LSTM (64 units) → dense head
Datasets	Sentiment140 (200,000 tweets sampled from 1.6 million, binary labels, binary cross-entropy) and the Twitter Sentiment Dataset (three classes, one-hot, MSE)
Independent variables	Optimiser ∈ {SGD, Adam, RMSProp} × learning rate ∈ {0.1, 0.01}
Dependent variables	Weighted F1 score on a held-out 20% split; epochs until validation accuracy stops improving (early stopping with best-weight restore)
Constants	Batch size 64, ReLU, 80/20 split, up to 50 epochs
Hardware	Google Colab Pro+, NVIDIA A100

Accuracy is reported as F1 rather than raw accuracy because the three-class dataset is imbalanced, so precision and recall were weighted by class frequency.

Results

F1 score

Optimiser	LR	Sentiment140 (binary)	Twitter (3-class, weighted)
SGD	0.1	0.792	0.960
SGD	0.01	0.789	0.970
Adam	0.1	0.668 (did not converge)	0.078 (did not converge)
Adam	0.01	0.786	0.976
RMSProp	0.1	0.668 (did not converge)	0.172 (did not converge)
RMSProp	0.01	0.792	0.988

Epochs to convergence

Optimiser	LR	Sentiment140	Twitter
SGD	0.1	19	35
SGD	0.01	47	43
Adam	0.01	1	19
RMSProp	0.01	1	18

Findings

• RMSProp at a learning rate of 0.01 was the best optimiser on both criteria: highest F1 on both datasets and the fastest convergence.
• Adaptive optimisers fail outright at a learning rate of 0.1. Adam and RMSProp both collapsed to predicting a single class (recall 1.0, precision 0.5 on the binary task) because the step size overshoots the minimum. SGD tolerated the higher rate and in fact converged faster with it.
• Adam and RMSProp converge an order of magnitude faster than SGD where they converge at all: one epoch versus 19 to 47 on Sentiment140.
• Data quality dominated. Every converged model scored far higher on the cleaner three-class dataset than on Sentiment140.

Limitations noted in the essay

Training is stochastic (initialisation and batch sampling), so each configuration was trained once and the curves were smoothed with an exponential moving average; repeated runs would tighten the comparison. Compute limited the study to 200,000 of the 1.6 million Sentiment140 tweets and two learning rates. Transformer models were out of reach on Colab's memory limits at the time.

Repository

extended-essay.pdf            the essay (3,830 words, 51 pages with appendix)
notebooks/sentiment140.ipynb  binary task: data prep, three models, training, plots, confusion matrices
notebooks/twitter.ipynb       three-class task, same structure
figures/                      convergence curves and confusion matrices rendered from the essay

The notebooks are transcribed from the essay's appendix. Datasets are downloaded from Kaggle and are not included.

Stack

Python, TensorFlow and Keras, NumPy, pandas, scikit-learn, Matplotlib, seaborn

---

Shahu Wagh · shahuwagh.com · github.com/Shahu-123