Struct llamapun::parallel_data::corpus::Corpus[][src]

pub struct Corpus {
    pub path: String,
    pub xml_parser: Parser,
    pub html_parser: Parser,
    pub tokenizer: Tokenizer,
    pub dnm_parameters: DNMParameters,
    pub extension: Option<String>,
}
Expand description

A parallel iterable Corpus of HTML5 documents

Fields

path: String

root directory

xml_parser: Parser

document XHTML5 parser

html_parser: Parser

document HTML5 parser

tokenizer: Tokenizer

DNM-aware sentence and word tokenizer

dnm_parameters: DNMParameters

Default setting for DNM generation

extension: Option<String>

Extension of corpus files (for specially tailored resources such as DLMF’s .html5) defaults to selecting .html AND .xhtml files

Implementations

Create a new parallel-processing corpus with the base directory dirpath

Get a parallel iterator over the documents

Trait Implementations

Returns the “default value” for a type. Read more

Auto Trait Implementations

Blanket Implementations

Gets the TypeId of self. Read more

Immutably borrows from an owned value. Read more

Mutably borrows from an owned value. Read more

Performs the conversion.

Performs the conversion.

The alignment of pointer.

The type for initializers.

Initializes a with the given initializer. Read more

Dereferences the given pointer. Read more

Mutably dereferences the given pointer. Read more

Drops the object pointed to by the given pointer. Read more

The type returned in the event of a conversion error.

Performs the conversion.

The type returned in the event of a conversion error.

Performs the conversion.