Backbone
classkeras_nlp.models.Backbone(*args, dtype=None, **kwargs)
Base class for all Backbone
models.
A Backbone
is the basic architecture for a given NLP model. Unlike a
keras_hub.models.Task
, a Backbone
is not tailored to any specific loss
function and training setup. A Backbone
generally outputs the last hidden
states of an architecture before any output predictions.
A Backbone
can be used in one of two ways:
Task
class, which will wrap and extend a Backbone
so it
can be used with high level Keras functions like fit()
, predict()
or
evaluate()
. Task
classes are built with a particular training
objective in mind (e.g. classification or language modeling).All backbones include a from_preset()
constructor which can be used to
load a pre-trained config and weights.
Example
# Load a BERT backbone with pre-trained weights.
backbone = keras_hub.models.Backbone.from_preset(
"bert_base_en",
)
# Load a GPT2 backbone with pre-trained weights at bfloat16 precision.
backbone = keras_hub.models.Backbone.from_preset(
"gpt2_base_en",
dtype="bfloat16",
trainable=False,
)
from_preset
methodBackbone.from_preset(preset, load_weights=True, **kwargs)
Instantiate a keras_hub.models.Backbone
from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset
can be passed as a
one of:
'bert_base_en'
'kaggle://user/bert/keras/bert_base_en'
'hf://user/bert_base_en'
'./bert_base_en'
This constructor can be called in one of two ways. Either from the base
class like keras_hub.models.Backbone.from_preset()
, or from
a model class like keras_hub.models.GemmaBackbone.from_preset()
.
If calling from the base class, the subclass of the returning object
will be inferred from the config in the preset directory.
For any Backbone
subclass, you can run cls.presets.keys()
to list
all built-in presets available on the class.
Arguments
True
, the weights will be loaded into the
model architecture. If False
, the weights will be randomly
initialized.Examples
# Load a Gemma backbone with pre-trained weights.
model = keras_hub.models.Backbone.from_preset(
"gemma_2b_en",
)
# Load a Bert backbone with a pre-trained config and random weights.
model = keras_hub.models.Backbone.from_preset(
"bert_base_en",
load_weights=False,
)
token_embedding
propertykeras_nlp.models.Backbone.token_embedding
A keras.layers.Embedding
instance for embedding token ids.
This layer embeds integer token ids to the hidden dim of the model.
enable_lora
methodBackbone.enable_lora(rank)
Enable Lora on the backbone.
Calling this method will freeze all weights on the backbone,
while enabling Lora on the query & value EinsumDense
layers
of the attention layers.
save_lora_weights
methodBackbone.save_lora_weights(filepath)
load_lora_weights
methodBackbone.load_lora_weights(filepath)
save_to_preset
methodBackbone.save_to_preset(preset_dir)
Save backbone to a preset directory.
Arguments