Lexical Analyzer
Lexical Analysis
Made with OpenOffice.org
1
The role of lexical analyzer: analyzer:
It is the first phase of a compiler. compiler. Its main task is to read the input characters & produce as output a sequence of tokens that the parser uses for syntax analysis. analysis. Upon receiving a “get next token” command from the parser, the lexical analyzer reads input characters until it can identify the next token. token. It may also perform certain secondary tasks at the user interface. interface. One such task is stripping out from the source program comments & white spaces in the form of blank, tab, & new – line characters. characters. Another is correlating error messages from the compiler with the source program. program.
Made with OpenOffice.org
2
The role of lexical analyzer (cont…): (cont…
It may keep track of the number of new – line characters seen, so that a line number can be associated with an error message. message. In some compilers, the lexical analyzer is in charge of making a copy of the source program with the error messages marked in it. it.
Made with OpenOffice.org
3
Issues in lexical analysis: analysis:
There are several reasons for separating the analysis phase of compiling into linear analysis and hierarchical analysis. analysis. Simpler design is perhaps the most important consideration. The separation of consideration. lexical analysis from syntax analysis often allows to simplify one or the other of these phases. phases. Compiler efficiency is improved. A separate lexical analyzer allows to construct a improved. specialized & potentially more efficient processor for the task. A large amount of time task. is spent reading the source program & partitioning it into tokens. Specialized tokens. buffering techniques for reading input characters & processing tokens can significantly speed up the performance of a compiler. compiler. Compiler portability is enhanced. Input alphabet peculiarities & other device– enhanced.
Made with OpenOffice.org
1
The role of lexical analyzer: analyzer:
It is the first phase of a compiler. compiler. Its main task is to read the input characters & produce as output a sequence of tokens that the parser uses for syntax analysis. analysis. Upon receiving a “get next token” command from the parser, the lexical analyzer reads input characters until it can identify the next token. token. It may also perform certain secondary tasks at the user interface. interface. One such task is stripping out from the source program comments & white spaces in the form of blank, tab, & new – line characters. characters. Another is correlating error messages from the compiler with the source program. program.
Made with OpenOffice.org
2
The role of lexical analyzer (cont…): (cont…
It may keep track of the number of new – line characters seen, so that a line number can be associated with an error message. message. In some compilers, the lexical analyzer is in charge of making a copy of the source program with the error messages marked in it. it.
Made with OpenOffice.org
3
Issues in lexical analysis: analysis:
There are several reasons for separating the analysis phase of compiling into linear analysis and hierarchical analysis. analysis. Simpler design is perhaps the most important consideration. The separation of consideration. lexical analysis from syntax analysis often allows to simplify one or the other of these phases. phases. Compiler efficiency is improved. A separate lexical analyzer allows to construct a improved. specialized & potentially more efficient processor for the task. A large amount of time task. is spent reading the source program & partitioning it into tokens. Specialized tokens. buffering techniques for reading input characters & processing tokens can significantly speed up the performance of a compiler. compiler. Compiler portability is enhanced. Input alphabet peculiarities & other device– enhanced.