PaliGemmaCausalLM
classkeras_hub.models.PaliGemmaCausalLM(preprocessor, backbone, **kwargs)
An end-to-end multi modal PaliGemma model for causal language modeling.
A causal language model (LM) predicts the next token based on previous tokens. This task setup can be used to train the model unsupervised on image and plain text input, or to autoregressively generate plain text similar to the data used for training.
This model has a generate()
method, which generates text based on a
prompt. The generation strategy used is controlled by an additional
sampler
argument on compile()
. You can recompile the model with
different keras_hub.samplers
objects to control the generation. By
default, "greedy"
sampling will be used.
This model can optionally be configured with a preprocessor
layer, in
which case it will automatically apply preprocessing to string inputs during
fit()
, predict()
, evaluate()
and generate()
. This is done by default
when creating the model with from_preset()
.
Arguments
keras_hub.models.PaliGemmaBackbone
instance.keras_hub.models.PaliGemmaCausalLMPreprocessor
or
None
. If None
, this model will not apply preprocessing, and
inputs should be preprocessed before calling the model.Examples
Use generate()
to do text generation.
image = np.random.rand(224, 224, 3)
pali_gemma_lm = keras_hub.models.PaliGemmaCausalLM.from_preset(
"pali_gemma_3b_mix_224"
)
pali_gemma_lm.generate(
{
"images": image,
"text": ["answer en where is the cow standing?\n"]
}
)
# Generate with batched prompts.
pali_gemma_lm.generate(
{
"images": [image, image],
"text": ["answer en where is the cow standing?\n", "caption en\n"]
}
)
Use generate()
without preprocessing.
image = np.random.rand(224, 224, 3)
inputs = {
"images": [image, image],
# Token ids for "<bos> Keras is".
"token_ids": np.array([[2, 214064, 603, 0, 0, 0, 0]] * 2),
# Use `"padding_mask"` to indicate values that should not be overridden.
"padding_mask": np.array([[1, 1, 1, 0, 0, 0, 0]] * 2),
}
pali_gemma_lm = keras_hub.models.PaliGemmaCausalLM.from_preset(
"pali_gemma_3b_mix_224",
preprocessor=None,
)
pali_gemma_lm.generate(inputs)
Custom backbone and vocabulary.
tokenizer = keras_hub.models.PaliGemmaTokenizer(
proto="proto.spm",
)
preprocessor = keras_hub.models.PaliGemmaCausalLMPreprocessor(
tokenizer=tokenizer,
sequence_length=128,
)
backbone = keras_hub.models.PaliGemmaBackbone()
pali_gemma_lm = keras_hub.models.PaliGemmaCausalLM(
backbone=backbone,
preprocessor=preprocessor,
)
from_preset
methodPaliGemmaCausalLM.from_preset(preset, load_weights=True, **kwargs)
Instantiate a keras_hub.models.Task
from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset
can be passed as
one of:
'bert_base_en'
'kaggle://user/bert/keras/bert_base_en'
'hf://user/bert_base_en'
'./bert_base_en'
For any Task
subclass, you can run cls.presets.keys()
to list all
built-in presets available on the class.
This constructor can be called in one of two ways. Either from a task
specific base class like keras_hub.models.CausalLM.from_preset()
, or
from a model class like keras_hub.models.BertTextClassifier.from_preset()
.
If calling from the a base class, the subclass of the returning object
will be inferred from the config in the preset directory.
Arguments
True
, saved weights will be loaded into
the model architecture. If False
, all weights will be
randomly initialized.Examples
# Load a Gemma generative task.
causal_lm = keras_hub.models.CausalLM.from_preset(
"gemma_2b_en",
)
# Load a Bert classification task.
model = keras_hub.models.TextClassifier.from_preset(
"bert_base_en",
num_classes=2,
)
Preset name | Parameters | Description |
---|---|---|
pali_gemma_3b_mix_224 | 2.92B | image size 224, mix fine tuned, text sequence length is 256 |
pali_gemma_3b_mix_448 | 2.92B | image size 448, mix fine tuned, text sequence length is 512 |
pali_gemma_3b_224 | 2.92B | image size 224, pre trained, text sequence length is 128 |
pali_gemma_3b_448 | 2.92B | image size 448, pre trained, text sequence length is 512 |
pali_gemma_3b_896 | 2.93B | image size 896, pre trained, text sequence length is 512 |
generate
methodPaliGemmaCausalLM.generate(
inputs, max_length=None, stop_token_ids="auto", strip_prompt=False
)
Generate text given prompt inputs
.
This method generates text based on given inputs
. The sampling method
used for generation can be set via the compile()
method.
If inputs
are a tf.data.Dataset
, outputs will be generated
"batch-by-batch" and concatenated. Otherwise, all inputs will be handled
as a single batch.
If a preprocessor
is attached to the model, inputs
will be
preprocessed inside the generate()
function and should match the
structure expected by the preprocessor
layer (usually raw strings).
If a preprocessor
is not attached, inputs should match the structure
expected by the backbone
. See the example usage above for a
demonstration of each.
Arguments
tf.data.Dataset
. If a
preprocessor
is attached to the model, inputs
should match
the structure expected by the preprocessor
layer. If a
preprocessor
is not attached, inputs
should match the
structure expected the backbone
model.sequence_length
of the
preprocessor
. If preprocessor
is None
, inputs
should be
should be padded to the desired maximum length and this argument
will be ignored.None
, "auto", or tuple of token ids. Defaults
to "auto" which uses the preprocessor.tokenizer.end_token_id
.
Not specifying a processor will produce an error. None stops
generation after generating max_length
tokens. You may also
specify a list of token id's the model should stop on. Note that
sequences of tokens will each be interpreted as a stop token,
multi-token stop sequences are not supported.backbone
propertykeras_hub.models.PaliGemmaCausalLM.backbone
A keras_hub.models.Backbone
model with the core architecture.
preprocessor
propertykeras_hub.models.PaliGemmaCausalLM.preprocessor
A keras_hub.models.Preprocessor
layer used to preprocess input.