KerasNLP Tokenizers
Tokenizers convert raw string input into integer input suitable for a Keras Embedding layer.
They can also convert back from predicted integer sequences to raw string output.
All tokenizers subclass keras_nlp.tokenizers.Tokenizer, which in turn
subclasses keras.layers.Layer. Tokenizers should generally be applied inside a
tf.data.Dataset.map
for training, and can be included inside a keras.Model for inference.
Tokenizer
WordPieceTokenizer
- WordPieceTokenizer class
- tokenize method
- detokenize method
- get_vocabulary method
- vocabulary_size method
- token_to_id method
- id_to_token method
SentencePieceTokenizer
- SentencePieceTokenizer class
- tokenize method
- detokenize method
- get_vocabulary method
- vocabulary_size method
- token_to_id method
- id_to_token method
BytePairTokenizer
- BytePairTokenizer class
- tokenize method
- detokenize method
- get_vocabulary method
- vocabulary_size method
- token_to_id method
- id_to_token method
ByteTokenizer
- ByteTokenizer class
- tokenize method
- detokenize method
- get_vocabulary method
- vocabulary_size method
- token_to_id method
- id_to_token method
UnicodeCodepointTokenizer
- UnicodeCodepointTokenizer class
- tokenize method
- detokenize method
- get_vocabulary method
- vocabulary_size method
- token_to_id method
- id_to_token method