← Back to search

Meaning of tokenize | Babel Free

Verb CEFR B2

/ˈtoʊ.kən.aɪz/

Definitions

To reduce to a token or set of tokens by lexical analysis.
transitive
To substitute sensitive data with meaningless placeholders.
transitive
To treat as a token minority.
transitive

CEFR level

B2

Upper Intermediate
This word is part of the CEFR B2 vocabulary — upper intermediate level.

See also

tokenized tokenizes tokenizing tokens tokes up toking