AdaptCodec: Content-Aware Speech Codec

Not all speech frames deserve the same number of bits.

AdaptCodec is a neural speech codec that dynamically allocates bitrate per frame based on content complexity. A lightweight classifier analyzes each frame's latent representation and decides how many RVQ codebook layers to use:

• Silence → 1 codebook (~0.3 kbps)
• Unvoiced consonants → 2 codebooks (~0.6 kbps)
• Simple voiced → 4 codebooks (~1.2 kbps)
• Complex voiced → up to 8 codebooks (~2.5 kbps)

The result: better quality-per-bit than fixed-rate codecs, or the same quality at lower average bitrate.

How It Works

Input wav ──► Encoder ──► Latent z ──┬──► Content Classifier ──► n_q per frame
                                     │
                                     └──► Adaptive RVQ (masked) ──► Decoder ──► Reconstructed wav

During training:

1. The encoder/decoder learn reconstruction (spectral + waveform losses)
2. The classifier learns from oracle labels (energy + pitch + ZCR features)
3. A bitrate penalty encourages the classifier to use fewer codebooks when possible
4. Soft sigmoid masking allows gradients to flow through the allocation decision

During inference, the classifier makes hard decisions per frame — no overhead beyond the tiny classifier forward pass.

Install

git clone https://github.com/DASS-057/AdaptCodec.git
cd AdaptCodec
pip install -e .

Quick Start

import torch
from adaptcodec import AdaptCodec

model = AdaptCodec()
wav = torch.randn(1, 1, 32000)  # 2s at 16kHz

# adaptive compression
tokens, n_q = model.compress(wav)
print(f"tokens: {tokens.shape}, avg codebooks: {n_q.float().mean():.1f}")

# decompress
wav_hat = model.decompress(tokens, n_q)

# check effective bitrate
print(f"bitrate: {model.average_bitrate(n_q):.2f} kbps")

Training

# Standard training with oracle labels
python scripts/train.py --config configs/base.yaml

# Low-bitrate target (~1.5 kbps average)
python scripts/train.py --config configs/low_bitrate.yaml

# With GAN for better perceptual quality
python scripts/train_gan.py --config configs/base.yaml --gan-start 30

Evaluation

python scripts/evaluate.py --test-dir data/test --checkpoint checkpoints/adaptcodec_ep200.pt

Visualization

See per-frame allocation on a test file:

python tools/visualize_allocation.py --wav test.wav --checkpoint checkpoints/adaptcodec_ep200.pt

Compare adaptive vs fixed-rate:

python tools/compare_fixed_vs_adaptive.py --wav test.wav --checkpoint checkpoints/adaptcodec_ep200.pt

Project Structure

AdaptCodec/
├── adaptcodec/
│   ├── codec.py          # Main model
│   ├── encoder.py        # Dilated residual encoder
│   ├── decoder.py        # Transposed conv decoder
│   ├── quantizer.py      # Adaptive RVQ with soft masking
│   ├── classifier.py     # Content classifier + oracle labeler
│   ├── discriminator.py  # Multi-period discriminator
│   ├── losses.py         # Spectral + bitrate penalty losses
│   ├── metrics.py        # PESQ, STOI, SNR with bitrate stats
│   └── dataset.py        # Audio dataset
├── configs/              # YAML training configs
├── scripts/              # Train, evaluate, encode/decode
├── tools/                # Visualization + comparison
├── analysis/             # Dataset analysis tools
└── tests/

License

Apache 2.0

Citation

@misc{adaptcodec2025,
  title={AdaptCodec: Content-Aware Neural Speech Codec with Adaptive Bitrate Allocation},
  author={瑾瑜},
  year={2025},
  url={https://github.com/DASS-057/AdaptCodec}
}