Whisper batch generation is not faster than loops about ctranslate2 HOT 5 OPEN

transcribe_with_vad (backends/init)

The transcribe_with_vad method in WhisperModel utilizes voice activity detection to transcribe audio files. It corrects the batch parameters for language codes, tasks, and initial prompts using fix_batch_param. The method then processes the audio files in batches through WhisperDataLoader, converting signals to mel spectrograms using LogMelSpectogram in self.preprocessor, and segments the audio based on voice activity. The transcription process is handled by generate_segment_batched, an abstract method to be implemented by subclasses. Progress is tracked using tqdm.

fix_batch_param (backends/init)

This function is utilized in the WhisperModel class for preparing batch parameters like language codes, tasks, and initial prompts, ensuring they match the number of audio files being processed.

Whisperdataloader (data)

The WhisperDataLoader class prepares audio data for transcription by segmenting and batching it. It relies on external configurations (SAMPLE_RATE, N_SAMPLES), utility functions (pad_or_trim, audio_batch_generator), and classes (BasicSegmenter, stitch_speech_segments from the same script, torch, numpy) for operation. It handles whether to use voice activity detection (speech_segmenter) or basic segmentation (basic_segmenter) based on input flags. The class segments audio files, optionally merges speech segments to respect maximum speech length, and creates batches that include processed audio signals, prompts, and metadata. The method data_collate_fn assembles these into a format suitable for model input, including padding or trimming audio to uniform length and organizing prompts. It supports dynamic time axis adjustment for batch processing and yields batches ready for transcription processing, tracking progress with tqdm.
LogMelSpectrogram (audio.py)

The LogMelSpectogram class, a subclass of nn.Module, is designed for converting audio signals into log-mel spectrogram features. It initializes with parameters for the mel-spectrogram calculation (n_mels, n_fft, hop_length, padding) and loads mel filter banks from a predefined file, registering them as a buffer. The class also contains an instance of TorchSTFT for performing short-time Fourier transforms (STFT). It provides a method get_seq_len to adjust sequence lengths based on the hop_length, and a forward method to apply padding (if required), compute the STFT, convert the power spectrogram to a mel scale using the loaded mel filters, apply logarithmic scaling, clip and scale the log-mel spectrograms. This process is essential for preparing audio data for deep learning models in speech processing tasks. It utilizes torch, numpy, F.pad from PyTorch's functional API, and custom configurations (N_MELS, N_FFT, HOP_LENGTH, BASE_PATH) for its operations.

WhisperModelCT2 (model.py)

WhisperModelCT2. This subclass provides a concrete implementation of the abstract method generate_segment_batched. Here's a brief overview of how WhisperModelCT2 handles the transcription process after reaching generate_segment_batched:
1.	Initialization and Configuration: The WhisperModelCT2 constructor initializes the model with configurations for ASR (Automatic Speech Recognition) options, loads the model and tokenizer, and sets up parameters for generating transcriptions.
2.	Model and Tokenizer Loading: It loads a translation or transcription model using ctranslate2 based on a path or model name. The tokenizer is also loaded from a specified file.
3.	Transcription Process (generate_segment_batched):
o	Converts the features (audio data processed into a suitable format like log-mel spectrograms) into a format expected by the ctranslate2 model.
o	Calls the ctranslate2 model's generate method with these features and the specified generation options. This step performs the actual ASR by generating text from the input audio features.
o	Decodes the output from the model into human-readable text using the loaded tokenizer.
o	Optionally calculates additional metrics like average log probability of the sequences and no-speech probability if specified in the generation options.
o	If word timestamps are required, it performs alignment of the generated text with the audio features to produce word-level timestamps. This involves calling align_words, which uses the ctranslate2 model's align function, and then assign_word_timings to assign timings to individual words.
4.	Word Timings and Alignment: If the option for word timestamps is enabled, WhisperModelCT2 uses the aligner_model (another instance of a ctranslate2 model) to align the words in the transcribed text with their corresponding positions in the audio. This process generates detailed timing information for each word in the transcription.
5.	Returning the Transcription: The method returns a list of dictionaries. Each dictionary contains the transcribed text and, depending on the configuration, may also include average log probabilities, no-speech probabilities, and word-level timing information.

model.transcribe_with_vad

model = whisper_s2t.load_model(model_identifier=model_identifier, backend='CTranslate2', device=self.device, compute_type=self.quantization, asr_options={'beam_size': self.beam_size}, cpu_threads=os.cpu_count())

MODEL TRANSCRIPTION PROCESS
----------------------------

model.transcribe_with_vad
│
├─ fix_batch_param (Adjustment of Parameters)
│  └─ Applies to: lang_codes, tasks, initial_prompts
│
└─ WhisperDataLoader (Data Preparation and Loading)
   │
   ├─ Conditional Branching: use_vad flag
   │  ├─ speech_segmenter (if VAD enabled)
   │  └─ basic_segmenter (if VAD disabled)
   │     └─ Optional: stitch_speech_segments (Merge if merge_chunks=True)
   │
   ├─ get_segmented_audio_signal (Audio Segmentation)
   │  ├─ tokenizer.sot_sequence (Start of Token Sequence)
   │  └─ tokenizer.encode (Encoding Initial Prompts)
   │
   ├─ data_collate_fn (Data Collation)
   │  ├─ pad_or_trim (Adjust Audio Signal Lengths)
   │  └─ External: Torch Operations (Stacking and Tensor Creation)
   │
   └─ Yields Batches to transcribe_with_vad (Transcription)
      │
      ├─ preprocessor (Feature Extraction)
      │  ├─ TorchSTFT (Spectral Transformation)
      │  └─ Mel Filter Application & Log Scaling
      │
      └─ generate_segment_batched (in WhisperModelCT2)
         │
         ├─ ctranslate2 Model's `generate` (ASR Generation)
         │  ├─ Decoding Output to Text
         │  ├─ Average Log Probability (Optional)
         │  └─ No-Speech Probability (Optional)
         │
         ├─ align_words (Word Alignment)
         │  ├─ aligner_model's `align` (Alignment Process)
         │  └─ assign_word_timings (Timestamp Assignment)
         │
         └─ Structured Transcription Output
            └─ Includes: Text, avg_logprob, no_speech_prob, word_timestamps
               └─ Creation of "out" variable (Result Packaging)

The difference between the original version of my transcriber and the new version that processes each file separately:

First Snippet: Processing Files Individually in a Loop
•	Sequential Processing: Each audio file is processed one at a time in a while loop, which continues until the file_queue is empty or enumeration_done is set. This approach allows for real-time updates and handling of files as they become available, which can be particularly useful in scenarios where files are being added to the queue dynamically.
Second Snippet: Batch Processing Multiple Files
•	Batch Processing: Processes a list of audio files (audio_files_str) in a single call to transcribe_with_vad. This approach is more efficient if all audio files are available at the start, as it can leverage batch processing optimizations.