lfeats

lfeats provides a unified interface to extract hidden representations from various speech foundation models such as HuBERT and Whisper. While these extracted features are task-independent, the package is primarily designed for speech generation tasks including text-to-speech and voice conversion.

Requirements

• Python 3.10+
• PyTorch 2.6.0+

Documentation

• Reference Manual

Installation

The latest stable release can be installed via PyPI:

pip install lfeats

Alternatively, the development version can be installed directly from the GitHub repository:

pip install git+https://github.com/sp-nitech/lfeats.git@master

Supported Models

Frame-Level Features

Model Name	Model Variant	Layers	Dimension	Paper	Source	Model Hub
contentvec	hubert-100	12	768	arXiv	GitHub
	hubert-500	12	768
data2vec	base	12	768	arXiv	GitHub
	large	24	1024
data2vec2	base	8	768	arXiv	GitHub
	large	16	1024
emotion2vec	base	8	768	arXiv	GitHub	🤗
emotion2vec+	seed	8	768			🤗
	base	8	768			🤗
	large	8	1024			🤗
hubert	base	12	768	arXiv	GitHub	🤗
	large	24	1024			🤗
	xlarge	48	1280			🤗
r-spin	wavlm-32	12	768	arXiv	GitHub
	wavlm-64	12	768
	wavlm-128	12	768
	wavlm-256	12	768
	wavlm-512	12	768
	wavlm-1024	12	768
	wavlm-2048	12	768
spidr	base	12	768	arXiv	GitHub
spin	hubert-128	12	768	arXiv	GitHub
	hubert-256	12	768
	hubert-512	12	768
	hubert-1024	12	768
	hubert-2048	12	768
	wavlm-128	12	768
	wavlm-256	12	768
	wavlm-512	12	768
	wavlm-1024	12	768
	wavlm-2048	12	768
sslzip	tiny	0	16	ISCA	GitHub	🤗
	base	0	256			🤗
unispeech-sat	base	12	768	arXiv	GitHub	🤗
	base+	12	768			🤗
	large	24	1024			🤗
w2v-bert2	base	24	1024	arXiv	GitHub	🤗
wav2vec2	base	12	768	arXiv	GitHub
	large	24	1024
	xlsr	24	1024	arXiv
	xlsr-v2	24	1024	arXiv	GitHub
wavlm	base	12	768	arXiv	GitHub	🤗
	base+	12	768			🤗
	large	24	1024			🤗
whisper	tiny	4	384	arXiv	GitHub	🤗
	base	6	512			🤗
	small	12	768			🤗
	medium	24	1024			🤗
	large	32	1280			🤗
	large-v2	32	1280			🤗
	large-v3	32	1280			🤗

Token-Level Features

Model Name	Model Variant	Hop Size [ms]	Dimension	Paper	Source	Model Hub
dacvae	base	40	128	arXiv	GitHub	🤗
higgs-audio	v2	40	1024	Blog	GitHub	🤗
mimi	base	80	512	arXiv	GitHub	🤗
x-codec	hubert	20	1024	arXiv	GitHub	🤗
	wavlm	20	1024			🤗

Utterance-Level Features

Model Name	Model Variant	Dimension	Paper	Source	Model Hub
ecapa-tdnn	base	192	arXiv	GitHub	🤗
next-tdnn	light	192	arXiv	GitHub
	base	192
	base-v2	192
r-vector	base	256	arXiv	GitHub	🤗
redimnet	b0	192	arXiv	GitHub
	b1	192
	b2	192
	b3	192
	b4	192
	b5	192
	b6	192
redimnet2	b0	192	arXiv	GitHub
	b1	192
	b2	192
	b3	192
	b4	192
	b5	192
	b6	192
redimnet2+	b6	192	arXiv	GitHub	🤗
wavlm-sv	base	512	arXiv	GitHub	🤗
	base+	512			🤗
x-vector	base	512	IEEE	GitHub	🤗

Important

Users must comply with the respective licenses of the models. Please refer to the original repositories for detailed licensing information.

Supported Resamplers

Resampler Type	Quality Preset	Source	License
lilfilter	base	GitHub	MIT
scipy	fft	Document	BSD 3-Clause
	poly	Document
soxr	quick	GitHub	LGPL v2.1+
	low
	medium
	high
	very-high
torchaudio	kaiser-fast	Document	BSD 2-Clause
	kaiser-best

Examples

Simple Usage

lfeats simplifies the process of extracting hidden states from various speech foundation models. You don't need to worry about differences between model types or input/output data types.

import lfeats
import numpy as np

# Prepare an audio waveform without zero-mean and unit-variance normalization.
# Either a NumPy array or a Torch tensor is accepted as the input of the extractor.
sample_rate = 16000
waveform = np.random.uniform(-1, 1, sample_rate)

# Initialize the extractor.
extractor = lfeats.Extractor(
    model_name="hubert",
    model_variant="base",
    resampler_type="torchaudio",
    resampler_preset="kaiser-best",
    device="cpu",
)

# Note: The model weights are automatically loaded during the first call to extractor(),
# so calling extractor.load() explicitly is optional.
extractor.load()

# Extract features.
features = extractor(waveform, sample_rate)
print(f"Shape: {features.shape}")  # (1, 50, 768)

# You can access the features as a NumPy array.
print(type(features.array))  # <class 'numpy.ndarray'>

# You can also access the features as a Torch tensor.
print(type(features.tensor))  # <class 'torch.Tensor'>

Layer Selection

lfeats allows you to extract features from specific layer(s). By default, the last layer is used.

import lfeats
import numpy as np

sample_rate = 16000
waveform = np.random.uniform(-1, 1, sample_rate)

extractor = lfeats.Extractor(model_name="hubert")

# Get the second-to-last layer output.
features = extractor(waveform, sample_rate, layers=-2)
print(f"Shape: {features.shape}")  # (1, 50, 768)

# Get the multiple layer outputs.
features = extractor(waveform, sample_rate, layers=(11, 12))
print(f"Shape: {features.shape}")  # (1, 50, 1536)

# Get all layer outputs as a concatenated vector.
features = extractor(waveform, sample_rate, layers="all")
print(f"Shape: {features.shape}")  # (1, 50, 9984)

Audio Chunking

To be computationally efficient and prevent mismatches between training and inference, long audio files can be processed by splitting them into chunks.

import lfeats
import numpy as np

sample_rate = 16000
waveform = np.random.uniform(-1, 1, 10 * sample_rate)

extractor = lfeats.Extractor(model_name="hubert")

# Processing a 10-second waveform with a 5-second chunk and 1-second overlap.
features = extractor(waveform, sample_rate, chunk_length_sec=5, overlap_length_sec=1)
print(f"Shape: {features.shape}")  # (1, 500, 768)

Sliding-Window Upsampling

Since the frame rate of speech foundation models is typically 20 ms, it often doesn't match the 5 ms requirement of speech generation tasks. lfeats bridges this gap by sliding the input waveform and interleaving the resulting features, providing a high-resolution output.

import lfeats
import numpy as np

sample_rate = 16000
waveform = np.random.uniform(-1, 1, sample_rate)

extractor = lfeats.Extractor(model_name="hubert")

# Extract features at a 5 ms frame shift.
features = extractor(waveform, sample_rate, upsample_factor=4)
print(f"Shape: {features.shape}")  # (1, 200, 768)

Utterance-Level Feature Extraction

lfeats can extract utterance-level features, e.g., speaker embeddings, as well as frame-level features.

import lfeats
import numpy as np

sample_rate = 16000
waveform = np.random.uniform(-1, 1, sample_rate)

extractor = lfeats.Extractor(model_name="ecapa-tdnn")

# The default aggregation method for utterance-level features is averaging.
features = extractor(waveform, sample_rate, overlap_length_sec=0, reduction="mean")
print(f"Shape: {features.shape}")  # (1, 1, 192)

Command-Line Interface

Once installed via pip, you can use the lfeats command directly from your terminal.

# Basic usage: extract features from a wav file
$ lfeats input.wav --output_format npz

# Process all audio files in a directory
$ lfeats input/dir --output_dir feats

# Process files listed in a file
$ lfeats input.scp --output_dir feats

# Specify model and layer
$ lfeats input.wav --model_name hubert --model_variant base --layers 12

Tip

For more details on all available flags and default values, simply run:

$ lfeats --help

License

This project is released under the Apache License 2.0.

Third-Party Licenses

lfeats partially incorporates the following repositories:

Repository	License
DAC-VAE	Apache-2.0
fairseq	MIT
NeXt-TDNN	Apache-2.0
R-Spin	MIT
S3PRL	Apache-2.0
SpeechBrain	Apache-2.0
Spin	MIT
TIMM	Apache-2.0