GAN-Augmented Low-Resource Speech Recognition

Generative speech augmentation for improving multilingual automatic speech recognition in low-resource languages.

GAN-Augmented Low-Resource Speech Recognition project preview

Project Overview

GAN-Augmented Low-Resource Speech Recognition investigates the challenge of training reliable automatic speech-recognition systems when only limited labelled speech data is available. Low-resource languages are particularly affected by data scarcity, making it difficult for modern ASR models to achieve the performance obtained for languages supported by substantially larger speech corpora.

The project addresses this limitation through generative data augmentation. GAN-based speech representations are used to increase the diversity of the available training data, while Wav2Vec2 serves as the downstream speech-recognition model. The augmented training pipeline was evaluated on multilingual speech data from Mozilla Common Voice 13.0, with Hindi and Tigrinya used to examine the approach across different low-resource settings.

Performance was evaluated using Word Error Rate by comparing recognition results before and after generative augmentation. The experiments produced an approximate 19% reduction in WER for Hindi and 18% for Tigrinya, demonstrating the potential of GAN-based augmentation to improve speech recognition when labelled training resources are constrained.

Technology Stack

  • Python
  • Generative Adversarial Networks (GANs)
  • Wav2Vec2
  • PyTorch
  • Hugging Face Transformers
  • Mozilla Common Voice 13.0
  • Speech Signal Processing
  • Automatic Speech Recognition

System Architecture

The system combines generative speech augmentation with transfer learning to expand low-resource training data and improve multilingual automatic speech recognition.

  1. Speech Dataset — provides labelled speech samples for the target low-resource languages.
  2. Audio Preprocessing — prepares and standardizes speech recordings for model training.
  3. GAN-Based Augmentation — generates additional speech representations to increase training diversity.
  4. Augmented Training Dataset — combines original and generated speech samples for downstream learning.
  5. Wav2Vec2 ASR Model — performs transfer learning and speech recognition using the expanded dataset.
  6. Evaluation Layer — measures recognition performance using Word Error Rate (WER).

System Workflow

  1. Low-resource speech samples are collected from the multilingual speech dataset.
  2. Audio recordings are preprocessed and prepared for model training.
  3. GAN-based augmentation is applied to generate additional speech representations.
  4. Original and generated samples are combined to construct an expanded training dataset.
  5. The Wav2Vec2 model is fine-tuned using the augmented speech data.
  6. The trained model performs automatic speech recognition on evaluation samples.
  7. Word Error Rate is calculated to compare baseline and GAN-augmented ASR performance.
  8. Performance improvements are evaluated separately for Hindi and Tigrinya.

Key Features

  • GAN-based augmentation for limited speech datasets.
  • Multilingual low-resource speech recognition.
  • Wav2Vec2-based transfer learning.
  • Support for Hindi and Tigrinya speech data.
  • Baseline and augmented-model performance comparison.
  • Word Error Rate-based quantitative evaluation.
  • End-to-end speech preprocessing, augmentation, training, and evaluation pipeline.

Results

  • Improved Wav2Vec2 recognition performance after incorporating GAN-generated speech representations.
  • Reduced Word Error Rate by approximately 19% for Hindi.
  • Reduced Word Error Rate by approximately 18% for Tigrinya.
  • Demonstrated the effectiveness of generative augmentation for multilingual ASR with limited labelled speech data.
  • Established an augmentation pipeline applicable to low-resource speech-recognition scenarios.