WhisperAudioConverter
classkeras_hub.layers.WhisperAudioConverter(
num_mels=80,
num_fft_bins=400,
stride=160,
sampling_rate=16000,
max_audio_length=30,
**kwargs
)
Whisper audio converter layer.
This layer takes in a batch of audio tensors, and computes the log-mel spectrogram features for each audio tensor.
The input audio tensor can either be of shape (length_of_audio,)
or
(batch_size, length_of_audio)
. The output is a tensor of shape
(batch_size, num_frames, num_mels)
, where num_frames
is
(max_audio_length * sampling_rate) / stride
.
Arguments
80
.400
.160
.16000
.max_audio_length * sampling_rate
. Defaults to 30
.Examples
audio_tensor = tf.ones((8000,), dtype="float32")
# Compute the log-mel spectrogram.
audio_converter = keras_hub.models.WhisperAudioConverter.from_preset(
"whisper_base_en",
)
audio_converter(audio_tensor)
# Compute the log-mel spectrogram for a batch of audio tensors.
audio_tensor_1 = tf.ones((8000,), dtype="float32")
audio_tensor_2 = tf.ones((10000,), dtype="float32")
audio_tensor = tf.ragged.stack([audio_tensor_1, audio_tensor_2], axis=0)
audio_converter(audio_tensor)
from_preset
methodWhisperAudioConverter.from_preset(preset, **kwargs)
Instantiate a keras_hub.layers.AudioConverter
from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset
can be passed as
one of:
'whisper_base_en'
'kaggle://user/whisper/keras/whisper_base_en'
'hf://user/whisper_base_en'
'./whisper_base_en'
You can run cls.presets.keys()
to list all built-in presets available
on the class.
This constructor can be called in one of two ways. Either from the base
class like keras_hub.models.AudioConverter.from_preset()
, or from a
model class like keras_hub.models.WhisperAudioConverter.from_preset()
.
If calling from the base class, the subclass of the returning object
will be inferred from the config in the preset directory.
Arguments
True
, the weights will be loaded into the
model architecture. If False
, the weights will be randomly
initialized.Examples
# Load an audio converter from a preset.
converter = keras_hub.layers.AudioConverter.from_preset(
"whisper_base_en"
)
# Convert some raw mono channel audio input.
converter(np.ones(2, 1_000))
Preset name | Parameters | Description |
---|---|---|
whisper_tiny_en | 37.18M | 4-layer Whisper model. Trained on 438,000 hours of labelled English speech data. |
whisper_base_en | 124.44M | 6-layer Whisper model. Trained on 438,000 hours of labelled English speech data. |
whisper_small_en | 241.73M | 12-layer Whisper model. Trained on 438,000 hours of labelled English speech data. |
whisper_medium_en | 763.86M | 24-layer Whisper model. Trained on 438,000 hours of labelled English speech data. |
whisper_tiny_multi | 37.76M | 4-layer Whisper model. Trained on 680,000 hours of labelled multilingual speech data. |
whisper_base_multi | 72.59M | 6-layer Whisper model. Trained on 680,000 hours of labelled multilingual speech data. |
whisper_small_multi | 241.73M | 12-layer Whisper model. Trained on 680,000 hours of labelled multilingual speech data. |
whisper_medium_multi | 763.86M | 24-layer Whisper model. Trained on 680,000 hours of labelled multilingual speech data. |
whisper_large_multi | 1.54B | 32-layer Whisper model. Trained on 680,000 hours of labelled multilingual speech data. |
whisper_large_multi_v2 | 1.54B | 32-layer Whisper model. Trained for 2.5 epochs on 680,000 hours of labelled multilingual speech data. An improved of whisper_large_multi . |