Preprocessor
classkeras_nlp.models.Preprocessor()
Base class for preprocessing layers.
A Preprocessor
layer provides a complete preprocessing setup for a
given task. It handles tokenization, audio/image conversion, and
any other necessary preprocessing steps.
This class can be subclassed similar to any keras.layers.Layer
, by
defining build()
, call()
and get_config()
methods. All subclasses
should set the tokenizer
or audio_converter
or image_converter
properties during construction as needed.
from_preset
methodPreprocessor.from_preset(preset, config_file="preprocessor.json", **kwargs)
Instantiate a keras_hub.models.Preprocessor
from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset
can be passed as
one of:
'bert_base_en'
'kaggle://user/bert/keras/bert_base_en'
'hf://user/bert_base_en'
'./bert_base_en'
For any Preprocessor
subclass, you can run cls.presets.keys()
to
list all built-in presets available on the class.
As there are usually multiple preprocessing classes for a given model,
this method should be called on a specific subclass like
keras_hub.models.BertTextClassifierPreprocessor.from_preset()
.
Arguments
Examples
# Load a preprocessor for Gemma generation.
preprocessor = keras_hub.models.GemmaCausalLMPreprocessor.from_preset(
"gemma_2b_en",
)
# Load a preprocessor for Bert classification.
preprocessor = keras_hub.models.BertTextClassifierPreprocessor.from_preset(
"bert_base_en",
)
save_to_preset
methodPreprocessor.save_to_preset(preset_dir)
Save preprocessor to a preset directory.
Arguments
tokenizer
propertykeras_nlp.models.Preprocessor.tokenizer
The tokenizer used to tokenize strings.