Tokenizing is the operation of splitting up a string of characters into a set of tokens. A token, in computing, is a segment of text, regardess whether it be readable or comprised of symbols. ...

The term is also used when, during the parsing of source code of some programming languages, the symbols are converted into another format that is much smaller. Most BASIC interpreters used this to save room, a command such as print would be replaced by a single number which uses much less room in memory. In fact most lossless compression systems use a form of tokenizing, although it's typically not referred to as such. Source code (commonly just source or code) is any series of statements written in some human-readable computer programming language. ... A programming language or computer language is a standardized communication technique for expressing instructions to a computer. ... BASIC (Beginners All-purpose Symbolic Instruction Code) is a family of high-level programming languages. ... Lossless data compression is a class of data compression algorithms that allow the original data to be reconstructed exactly from the compressed data. ...

In human cognition tokenization is often used to refer to the process of converting a sensory stimulus into a cognitive "token" suitable for internal processing. A stimulus that is not correctly tokenized may not be processed or may be incorrectly merged with other stimuli.