Module llamapun::util::data_helpers

Helpers with transactional logic related to llamapun::data which doesn’t fit with the main structs TODO: May be reorganized better with some more thought, same as path_helpers

Structs

LexicalOptions

Options for lexical normalization on an individual word

Functions

ams_normalize_word_range

Normalization of word lexemes created for the “AMS paragraph classification” experiment operating on a DNMRange representation

heading_from_node_aux

Provides a string for a given heading node, using DNM-enabled word-tokenization TODO: This is a low-level auxiliary function, we may need to build more user-facing interfaces if it becomes more widely useful

invalid_for_english_latin

Check if the given DNM contains valid English+Latin content

normalize_heading_title

Attempt to recover the “type” of a potentially specialized heading, e.g. “definition xiii a”->“definition”