FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates

Abstract

Neural audio codecs are foundational to speech language models. It is expected to have a low frame rate and decoupled semantic and acoustic information. A lower frame rate codec can reduce the computational cost of speech language models by shortening the sequence length. Recent studies have developed 12.5Hz low-frame-rate audio codecs, but even lower frame rate codecs remain underexplored. We find that a major challenge for very low frame rate tokens is missing semantic information. This paper introduces FlexiCodec to address this limitation. FlexiCodec improves semantic preservation with a dynamic frame rate approach and introduces a novel architecture featuring an ASR feature-assisted dual stream encoding and Transformer bottlenecks. With dynamic frame rates, it uses less frames at information-sparse regions through adaptively merging semantically similar frames. A dynamic frame rate also allows FlexiCodec to support inference-time controllable frame rates between 3Hz and 12.5Hz. Experiments on 6.25Hz, 8.3Hz and 12.5Hz average frame rates confirm that FlexiCodec excels over baseline systems in semantic information preservation and delivers a high audio reconstruction quality. We also validate the effectiveness of FlexiCodec in language model-based TTS.

Method

Examining the impact of very low frame rates

We first investigate the performance of representative audio codecs at very low frame rates. We created three new baseline versions by retraining DAC and DualCodec to operate at 12.5Hz, 8.3Hz, and 6.25Hz, respectively.

Semantic Information Preservation

The following audios are the codec-reconstructed audios using their RVQ-1 tokens. For these RVQ-1 audios, we focus on their intelligibility instead of the acoustic details, because at such low frame rate and bitrate, it is not possible to reconstruct the acoustic details, but we can maintain the core semantic information that are useful for downstream models.

The audios above show that FlexiCodec audios are more intelligible than other codecs especially at lower frame rates like 6.25Hz. This is confirmed by using a speech recognition to compute the WERs of the audios, as illustrated below. The rich semantic information of FlexiCodec RVQ-1 tokens is helpful for the downstream models (especially the AR models) to generate more intlligible speech.

Acoustic Quality Evaluation

Audio codecs can use more RVQ layers to reconstruct acoustic details. The following audios are reconstructed audios using 8 RVQ layers for each codec variant. You may not easily hear the difference, so we attach a figure of the PESQ metric.

The results show that acoustic quality metrics show more moderate differences across systems and frame rates, compared to the dramatic differences in the previous semantic evaluation. We think that the acoustic fidelity is more constrained by bitrate, and because the bitrates of the three systems are at the same level, the difference is not pronounced.

Additional Codec-Reconstructed Audio Comparisons

The following audio comparisons showcase the quality differences between FlexiCodec and other open-source codec systems. These samples demonstrate how FlexiCodec performs compared to established codec architectures at different compression rates.

Text-to-Speech

We built a text-to-speech system with FlexiCodec. This system is an AR + NAR(flow matching) TTS. The AR predicts each FlexiCodec token and its frame length using two parallel prediction heads. Because the AR model is usually the slowest part in TTS, it can greatly benefit from FlexiCodec's low frame rate. For the NAR, we tried two schemes, one is using the 50Hz male spectrogram feature, the second one is using the 12.5Hz FlexiCodec feature.

We find that FlexiCodec-TTS delivers strong quality at less compute than baselines. With 6.25Hz AR + 50Hz NAR, it matches or surpasses CosyVoice and SparkTTS. A 50Hz NAR consistently outperforms 12.5Hz across WER, SIM-0, NMOS, and QMOS, showing that higher temporal resolution is important for NAR. Lower AR rates (8.3/6.25Hz) preserve or slightly improve accuracy with 50Hz NAR—likely due to better semantic-acoustic disentanglement and shorter sequences that simplify attention.

Frame rate controllability analysis

This section demonstrates FlexiCodec's frame rate controllability through different merging threshold settings. The threshold parameter controls the dynamic frame rate mechanism, where lower thresholds lead to more aggressive frame merging and lower effective frame rates. As can be heard from the audios. the intelligibility is adequate at > 4.5Hz, but becomes unusable for 3.6Hz and 3.0Hz.