Meaning of mistokenize | Babel Free

Definitions

To tokenize incorrectly.

Examples

“C++ considers ::*, .* and ->* each to be a single token and a single operator. Some pre-Release 2.0 implementations mistokenize expressions involving pointer-to-pointer-to-member.”

“These were mostly proper names, such as Ronny Johnsen, or foreign language items such as ambre solaire (French) and fairie queene (Middle English), as well as a few misspelt or mistokenized items.”

“The sentences were tokenized into words using the regex tokenizer which avoided the problems of mistokenizing while using the default NLTK tokenizer.”

“Similarly, all other two-character atomic representations in SMILES are being mistokenized.”

CEFR level