What Is A Tokenizer

[Solved] What Is A Tokenizer | Solidity - Code Explorer | yomemimo.com
Question : what is a tokenizer

Answered by : vast-vicua-qiks2revxjd1

A tokenizer breaks a stream of text into tokens, usually by looking for whitespace (tabs, spaces, new lines).
A lexer is basically a tokenizer, but it usually attaches extra context to the tokens -- this token is a number, that token is a string literal, this other token is an equality operator.
A parser takes the stream of tokens from the lexer and turns it into an abstract syntax tree representing the (usually) program represented by the original text.

Source : | Last Update : Mon, 07 Feb 22

Answers related to what is a tokenizer

Code Explorer Popular Question For Solidity