Struct llamapun::dnm::DNMParameters

pub struct DNMParameters {
    pub special_tag_name_options: HashMap<String, SpecialTagsOption>,
    pub special_tag_class_options: HashMap<String, SpecialTagsOption>,
    pub normalize_white_spaces: bool,
    pub wrap_tokens: bool,
    pub normalize_unicode: bool,
    pub stem_words_once: bool,
    pub stem_words_full: bool,
    pub convert_to_lowercase: bool,
    pub support_back_mapping: bool,
}

Parameters for the DNM generation

Fields

special_tag_name_options: HashMap<String, SpecialTagsOption>

How to deal with special tags (e.g. <math> tags)

special_tag_class_options: HashMap<String, SpecialTagsOption>

How to deal with tags with special class names (e.g. ltx_note_mark) Remark: If both a tag name and a tag class match, the tag name rule will be applied.

normalize_white_spaces: bool

merge sequences of whitespaces into a single ’ ’. Doesn’t affect tokens

wrap_tokens: bool

put spaces before and after tokens

normalize_unicode: bool

Replace unicode characters by the ascii code representation

stem_words_once: bool

Apply the morpha stemmer once to the text nodes

stem_words_full: bool

Apply the morpha stemmer to the text nodes as often as it changes something

convert_to_lowercase: bool

Move to lowercase (remark: The stemmer does this automatically)

support_back_mapping: bool

Support back mapping, i.e. mapping plaintext offsets back to the DOM

Implementations

impl DNMParameters

pub fn llamapun_normalization() -> DNMParameters

Normalize in a reasonable way for our math documents

pub fn check(&self)

Prints warnings, if the parameter settings don’t make sense. Doesn’t check for every possible stupidity

Trait Implementations

impl Clone for DNMParameters

fn clone(&self) -> DNMParameters

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DNMParameters

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for DNMParameters

fn default() -> DNMParameters

Don’t do anything fancy and specific by default