Module llamapun::tokenizer

source · [−]
Expand description

Provides functionality for tokenizing sentences and words

Structs

Tokenizer
Stores auxiliary resources required by the tokenizer so that they need to be initialized only once