An implementation of the relative attention mechanism from Huang et al.'s "Music Transformer" (ICLR 2019), trained on the JSB Chorales dataset, with an ablation against standard absolute-position attention. On top of that, I use the trained model to harmonize a melody it never saw anything like during training — a Lebanese/maqam-influenced tune I recorded on piano — to see what happens when a model that only knows Western tonal harmony is asked to accompany something outside that tradition.
Regular self-attention has no built-in sense of how far apart two positions are — it only sees content, not distance. You can add that back in with relative position embeddings (Shaw et al. 2018), but the naive way to do it needs a full (L, L, D) tensor of position embeddings, which is quadratic in sequence length and gets expensive fast for anything more than a few hundred steps.
Huang et al.'s trick: instead of materializing that full tensor, matmul queries against a compact (L, D) relative embedding table, then use a pad/reshape/slice operation ("skewing") to turn the resulting (query_pos, distance) matrix into the (query_pos, key_pos) matrix attention actually needs. Memory drops from O(L^2 D) to O(LD). src/relative_attention.py implements this (skew() + RelativeGlobalAttention), with a self-test that checks the skew against a hand-worked 4x4 example before trusting it inside the model.
Trained two versions of the same model on JSB Chorales — one with relative attention, one with plain causal attention + absolute sinusoidal position encodings — everything else identical (5 layers, 256-dim, 8 heads, same data, same training budget). This reproduces the paper's own ablation (their Table 2):
| Paper (nats/token) | This run | |
|---|---|---|
| Baseline (absolute position) | 0.417 | 0.635 |
| Relative attention | 0.357 | 0.542 |
| Improvement | 0.060 | 0.093 |
Absolute numbers are higher than the paper's — smaller model, less tuning, a 4GB GTX 1050 Ti instead of whatever Google Brain trained on — but relative attention wins by roughly the same margin, which is the result that actually matters here.
The paper's architecture already supports melody-conditioned generation (they demonstrate it on Maestro with a seq2seq setup). JSB Chorales makes this cheap to test without retraining: the model is trained on interleaved soprano/alto/tenor/bass tokens, so harmonizing a given melody is just constrained decoding — force the real melody into every soprano slot, let the model sample alto/tenor/bass as usual, conditioned on everything so far.
So I recorded a few maqam-influenced melodies on piano, transcribed them to pitch sequences (scripts/transcribe_melody.py, using librosa's pYIN pitch tracker), and fed them into the harmonizer (src/harmonize.py).
Two things had to be checked before trusting any result here:
Is the transcription accurate? Rendered a real Bach soprano line to audio at a known tempo, ran it back through the transcription pipeline, and compared against the true pitches. 100% exact match on that test (scripts/validate_transcription.py). Separately, transcribing the actual Bach melody used as the control for the harmonization comparison came back 99.5% exact-match (207/208 steps).
Does that transcription noise explain the harmonization quality difference? Harmonized the same Bach melody two ways — straight from its known pitches, and via the transcription pipeline — under identical sampling settings. The transcription-based version sounds close to, but a bit behind, the ground-truth version. Some real degradation from transcription noise, but nowhere close to the gap between either Bach version and the maqam harmonization.
That gap is the actual finding: harmonizing the maqam melody comes out noticeably more dissonant and less coherent than harmonizing a Bach melody under the same settings, even with transcription ruled out as the cause. The model's Western tonal-harmony assumptions, learned entirely from four-part Bach chorales, don't transfer cleanly to melodic material built on a different scale system.
One limitation worth stating plainly: standard MIDI has 12 semitones per octave, so any quarter-tone inflection in the original maqam playing gets rounded to the nearest semitone during transcription. That's a ceiling of the representation itself, not something the transcription pipeline could fix — I don't have room to represent microtonality with a model trained on MIDI-derived tokens.
I also couldn't find an existing symbolic (MIDI) dataset for Arabic maqam music to compare against or train on — recording my own melodies sidesteps that gap, though it does mean the "melody outside the training distribution" side of this comparison is a handful of short clips rather than a full dataset.
src/
relative_attention.py skew() + RelativeGlobalAttention, self-tested
model.py decoder-only transformer, relative or baseline attention
data.py JSB Chorales -> token sequences
train.py training loop, AMP, early stopping
generate.py checkpoint -> generated chorale -> MIDI
harmonize.py constrained decoding: given melody -> full 4-voice harmonization
scripts/
download_data.py fetch JSB Chorales
explore_data.py inspect the raw data format
export_real_chorale.py dataset chorale -> MIDI (unmodified, for comparison)
export_melody.py export just a melody line -> MIDI
transcribe_melody.py audio -> pitch sequence (librosa pYIN)
validate_transcription.py checks transcription accuracy against known ground truth
compare_pitches.py diff a transcribed pitch sequence against a known-correct one
midi_to_wav.py renders MIDI to audio with a small built-in synthesizer (no external soundfont/binary needed)
samples/
reproduction/ free generation from the trained model, no melody conditioning
extension/
bach_control/ Bach melody harmonized from its exact dataset pitches
bach_control_transcribed/ same melody, but round-tripped through transcription first
my_melody_transcribed/ the maqam melody, transcribed and harmonized
validation/ transcription accuracy check against known ground truth
recordings/ raw audio recordings used as harmonization input
checkpoints/ trained model weights (relative, baseline, and the one used for generation/harmonization)
python -m venv venv
.\venv\Scripts\Activate.ps1
pip install torch --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
python scripts\download_data.py
If you don't have a CUDA GPU, install plain torch instead of the cu121 build — everything still runs on CPU, just slower for training.
Train (run both variants to reproduce the ablation):
python src\train.py --variant relative
python src\train.py --variant baseline
Generate a new chorale from scratch:
python src\generate.py --steps 512 --top_p 0.7 --humanize --out sample.mid
Harmonize a melody pulled from the dataset (sanity check) or your own transcribed audio:
python src\harmonize.py --melody_split valid --melody_idx 0 --melody_steps 32 --out harmonized.mid
python src\harmonize.py --melody_file my_pitches.txt --top_p 0.7 --humanize --out harmonized.mid
Transcribe a monophonic recording into a pitch sequence:
python scripts\transcribe_melody.py --audio my_melody.wav --bpm 90 --out my_pitches.txt
Render any MIDI file to a .wav you can actually listen to:
python scripts\midi_to_wav.py --midi samples\reproduction\sample.mid --out sample.wav
- The ~0.4% of JSB Chorales timesteps with fewer than 4 sounding voices get REST-padded rather than assigned to a specific voice, since the source data doesn't label which voice is resting. Documented approximation, not guaranteed-correct.
- JSB Chorales is a fixed 16th-note grid with no expressive micro-timing, so generated output is somewhat rhythmically rigid by construction — that's true of real Bach chorales too, not an artifact of this implementation.
- Model is smaller and trained for less time than the paper's (4GB VRAM locally), so absolute loss values are higher even though the relative-vs-baseline comparison holds up.
- Maqam melody transcription is limited to semitone resolution; microtonal inflections in the original playing aren't representable in standard MIDI pitch numbers.
- No symbolic maqam dataset exists to train or evaluate against directly, so the extension is based on a small number of self-recorded melodies rather than a full dataset.
Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C., Dai, A. M., Hoffman, M. D., Dinculescu, M., & Eck, D. (2019). Music Transformer: Generating Music with Long-Term Structure. ICLR 2019. https://arxiv.org/abs/1809.04281