Back to ComputerTerms, InformationRetrieval
Lexical analysis is the process of converting an input stream of characters into a stream of words or tokens. Tokens are groups of characters with collective significance. This is the first stage of automated indexing and of the query processing.
Back to ComputerTerms, InformationRetrieval