Few-Shot Transfer for Speech Enhancement Using SEGAN with Stability Guardrails

Downloads

Authors

  • Rubi Sharma Rajiv Gandhi University, India
  • Firos A. Rajiv Gandhi University, India

Abstract

High-quality speech communication is often compromised by background noise, reducing intelligibility and perceived quality.We investigate data-efficient few-shot transfer of the speech enhancement generative adversarial network (SEGAN) to a new noise domain. Starting from a generator pre-trained on VoiceBank–DEMAND, we adapt the model to MiniLibriMix using only 300 paired noisy–clean examples. To prevent overfitting and catastrophic forgetting, we introduce stable adversarial few-shot enhancement (SAFE), a three-fold stabilisation strategy with (1) exponential-moving-average (EMA) weight averaging, (2) L2-SP weight anchoring to the source-domain parameters, and (3) a teacher–student consistency loss. SAFE maintains VoiceBank performance (PESQ ≈ 1.84; STOI ≈ 90 %) and, after an optional perceptual fine-tuning stage (L1 + MR-STFT), yields substantial target-domain gains on MiniLibriMix (PESQ 1.11 → 1.26, STOI 71.5 % → 81.5 %) with only a minor source-domain trade-off in STOI. Ablation experiments demonstrate that EMA provides the strongest stabilising effect, while L2-SP and consistency regularisation offer complementary benefits. These results suggest that stable few-shot adaptation may make lightweight time-domain speech enhancers practical for rapid deployment in novel acoustic environments.

Keywords:

speech enhancement, generative adversarial networks, few-shot learning, transfer learning, domain adaptation, stability regularization

References


  1. Barker J., Marxer R., Vincent E., Watanabe S. (2015), The third ‘CHiME’ speech separation and recognition challenge: Dataset, task and baselines, [in:] 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), https://doi.org/10.1109/ASRU.2015.7404837

  2. Cao R., Abdulatif S., Yang B. (2022), CMGAN: Conformer-based metric GAN for speech enhancement, [in:] Proceedings Interspeech 2022, pp. 936–940, https://doi.org/10.21437/Interspeech.2022-517

  3. Hou N., Xu C., Chng E.S., Li H. (2019), Domain adversarial training for speech enhancement, [in:] 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), https://doi.org/10.1109/APSIPAASC47483.2019.9023218

  4. Kim D., Chung S.W., Han H., Ji Y, Kang H.G. (2023), HD-DEMUCS: General speech restoration with heterogeneous decoders, [in:] Proceedings Interspeech 2023 , https://doi.org/10.21437/Interspeech.2023-1642

  5. Kim S., Athi M., Shi G., Kim M., Kristjansson T. (2024), Zero-shot test-time adaptation via knowledge distillation for personalized speech denoising and dereverberation, The Journal of the Acoustical Society of America, 155(2): 1353–1367, https://doi.org/10.1121/10.0024621

  6. Kim S., Kim M. (2021), Test-time adaptation toward personalized speech enhancement: Zero-shot learning with knowledge distillation, [in:] 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), https://doi.org/10.1109/WASPAA52581.2021.9632771

  7. Kingma D.P., Ba J. (2015), Adam: A method for stochastic optimization, [in:] International Conference on Learning Representations (ICLR).

  8. Li L., Wudamu, Kurzinger L., Watzel T., Rigoll G. (2021), Lightweight end-to-end speech enhancement GAN using sinc convolutions, Applied Sciences, 11(16): 7564, https://doi.org/10.3390/app11167564

  9. Li X., Grandvalet Y., Davoine F. (2018), Explicit inductive bias for transfer learning with convolutional networks (L2-SP), arXiv, https://doi.org/10.48550/arXiv.1802.01483

  10. Liao C.-F., Tsao Y., Lee H.-Y., Wang H.-M. (2019), Noise adaptive speech enhancement using domain adversarial training, [in:] Proceedings Interspeech 2019, https://doi.org/10.21437/Interspeech.2019-1519

  11. Loshchilov I., Hutter F. (2016), SGDR: Stochastic gradient descent with warm restarts, arXiv, https://doi.org/10.48550/arXiv.1608.03983

  12. Lv R. et al. (2024), SASEGAN-TCN: Speech enhancement algorithm based on self-attention GAN and temporal convolutional network, Mathematical Biosciences and Engineering, 21(3): 3860–3875, https://doi.org/10.3934/mbe.2024172

  13. Pascual S., Bonafonte A., Serra J. (2017), SEGAN: Speech enhancement generative adversarial network, [in:] Proceedings Interspeech 2017, https://doi.org/10.21437/Interspeech.2017-1428

  14. Pascual S., Park M., Serra J., Bonafonte A., Ahn K.-H. (2018), Language and noise transfer in speech enhancement generative adversarial network, [in:] 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), https://doi.org/10.1109/ICASSP.2018.8462322

  15. Reddy C.K.A. et al. (2020), The INTERSPEECH 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results, [in:] Proceedings Interspeech 2020, pp. 2492–2496, https://doi.org/10.21437/Interspeech.2020-3038

  16. Rix A.W., Beerends J.G., Hollier M.P., Hekstra A.P. (2001), Perceptual evaluation of speech quality (PESQ) – A new method for speech quality assessment, [in:] 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221), https://doi.org/10.1109/ICASSP.2001.941023

  17. Shi H., Mimura M., Wang L., Dang J., Kawahara T. (2023), Time-domain speech enhancement assisted by multiresolution spectrograms, arXiv, https://doi.org/10.48550/arXiv.2303.14593

  18. Taal C.H., Hendriks R.C., Heusdens R., Jensen J. (2011), A short-time objective intelligibility measure for time-frequency weighted noisy speech, [in:] 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, https://doi.org/10.1109/ICASSP.2010.5495701

  19. Tarvainen A., Valpola H. (2017), Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results, arXiv, https://doi.org/10.48550/arXiv.1703.01780

  20. Valentini-Botinhao C. (2017), Noisy speech database for training speech enhancement algorithms and TTS models, University of Edinburgh. School of Informatics. Centre for Speech Technology Research (CSTR), https://doi.org/10.7488/ds/2117

  21. Vinotha R., Hepsiba D., Vijay Anand L.D., Andrew J., Eunice R.J. (2024), Enhancing dysarthric speech recognition through SepFormer and hierarchical attention network models with multistage transfer learning, Scientific Reports, 14(1): 29455, https://doi.org/10.1038/s41598-024-80764-w

  22. Wang S., Li W., Siniscalchi S.M., Lee C.-H. (2020), A cross-task transfer learning approach to adapting deep SE models to unseen background noise using paired senone classifiers, [in:] ICASSP 2020 – 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), https://doi.org/10.1109/ICASSP40776.2020.9054543

  23. Yamamoto R., Song E., Kim J.-M. (2020), Parallel WaveGAN: A fast waveform generation model based on GANs with multi-resolution spectrogram, [in:] ICASSP 2020 – 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), https://doi.org/10.1109/ICASSP40776.2020.9053795

  24. Yang Y., Trigoni N., Markham A. (2024), Pre-training feature-guided diffusion model for speech enhancement, arXiv, https://doi.org/10.48550/arXiv.2406.07646

  25. Yu C., Fu S.-W., Hsieh T.-A., Tsao Y., Ravanelli M. (2021), OSSEM: One-shot speaker adaptive speech enhancement using meta-learning, arXiv, https://doi.org/10.48550/arXiv.2111.05703

  26. Yu J., Chen H., Luo Y., Gu R., Weng C. (2023), High fidelity speech enhancement with band-split RNN, [in:] Proceedings Interspeech 2023, pp. 2483–2487, https://doi.org/10.21437/Interspeech.2023-1433

  27. Zhang C. et al. (2023), A survey on audio diffusion models: Text-to-speech synthesis and speech enhancement, arXiv, https://doi.org/10.48550/arXiv.2303.13336