Scope note

Text

Following the standard definition in Natural Language Processing, a token is the basic unit of text created by splitting a string into smaller segments, such as words, subwords, or characters. This process, known as tokenization, converts raw text into a structured format that algorithms can numerically encode and process. Tokens serve as the fundamental building blocks for models to understand syntax, semantics, and context within a given language.

Language

en