Struct llamapun::ngrams::Ngrams

pub struct Ngrams {
    pub anchor: Option<String>,
    pub window_size: usize,
    pub n: usize,
    pub counts: HashMap<String, usize>,
}

Ngrams are dictionaries with

Fields

anchor: Option<String>

anchor word that must be present in all ngram contexts (in their window)

window_size: usize

if an anchor word is given, word window size, applied to the left and to the right of the anchor word

n: usize

n-grams for a sequence of n words

counts: HashMap<String, usize>

statistics hashmap for the occurence counts

Implementations

impl Ngrams

pub fn get(&self, word: &str) -> usize

Get the word count

pub fn insert(&mut self, phrase: String)

count a newly seen ngram phrase

pub fn sorted(&self) -> Vec<(&String, usize)>

obtain the ngram report, sorted by descending frequency

pub fn distinct_count(&self) -> usize

get the number of distinct ngrams recorded

pub fn add_content(&mut self, content: &str)

add content for ngram analysis, typically a paragraph or a line of text

pub fn add_anchored_content(&mut self, content: &str)

In essence, for a given window size W, a word at index i is justified to participate in the ngrams if there is an instance of an anchor word in the range of words [i-W, i+W]. this can be highly irregular e.g. “word word anchor word anchor word word”, so we record flexibly looking for no-justification cutoffs, where a continuous word sequence is recorded for ngram counts

pub fn record_words(&mut self, words: Vec<&str>)

Take an arbitrarily long vector of words, and record all (overlapping) ngrams obtainable from it

Trait Implementations

impl Default for Ngrams

fn default() -> Ngrams

Returns the “default value” for a type.