JDT Core / HowTo: Generate the Parser
Where to get the parser generator
The parser files and resources are automatically generated using the LPG parser generator. This tools has been renamed Jikes Parser generator. You can find more information and latest releases at this link. The latest tool is provided in source format. We don't provide any help for compiling these source files. Refer to the link above if you have trouble to get binaries.
Our grammar is generated using the version 2.30 of LPG. If newer versions fail to generate resources from our grammar, please send request to the Jikes Parser Generator team.

Where to get the grammar
The latest grammar is always located in the grammar() method of the Parser class. Go to the org.eclipse.jdt.core plugins directory (eclipse\plugins\org.eclipse.jdt.core where eclipse is the root of your eclipse installation) and open the jdtcoresrc.zip file. Then search for the Parser.java inside the folder org\eclipse\jdt\internal\compiler\parser\. You need to look for the method named grammar(). Then copy its contents from:
--main options
%options ACTION, AN=JavaAction.java, GP=java, 
....
$end
-- need a carriage return after the $end
into a file called java.g. It is important to add a carriage return at the end of the last line. You can save this file where you want, we will assume from thereon you saved it in d:\temp\.

What to do with these files and update the parser class...
Assuming, the LPG executable (lpg.exe or jikespg.exe) is located inside d:\lpg.
  1. First in a console, run:
     	d:
     	cd \temp
     	d:\lpg\lpg.exe java.g
     
  2. You will get an output that looks like this:
     LPG Parser Generator (V2.30)            Tue Apr 02 12:49:13 2002
    %OPTIONS ACTION, AN=JavaAction.java, GP=java,
    %OPTIONS FILE-PREFIX=java, ESCAPE=$, PREFIX=TokenName, OUTPUT-SIZE=125 ,
    %OPTIONS NOGOTO-DEFAULT, SINGLE-PRODUCTIONS, LALR=1 , TABLE=TIME ,
    %OPTIONS ERROR_MAPS
    %OPTIONS first follow
    %OPTIONS TRACE=FULL ,
    %OPTIONS VERBOSE
    Options in effect:
        ACTION  ACTFILE-NAME=JavaAction.java  BLOCKB=/.  BLOCKE=./  BYTE  CONFLIC
        DEFAULT=5  NODEBUG  DEFERRED  NOEDIT  ERROR-MAPS  ESCAPE=$
        FILE-PREFIX=java  FIRST  FOLLOW  GENERATE-PARSER=JAVA  NOGOTO-DEFAULT
        HACTFILE-NAME=javahdr.java  HBLOCKB=/:  HBLOCKE=:/  LALR=1  LIST
        MAX-DISTANCE=30  MIN-DISTANCE=3  NAMES=OPTIMIZED  NONT-CHECK  ORMARK=|
        OUTPUT-SIZE=125  PREFIX=TokenName  READ-REDUCE  NOSCOPES  NOSHIFT-DEFAULT
        SINGLE-PRODUCTIONS  STACK-SIZE=128  STATES  SUFFIX=  TABLE=TIME  TRACE=FU
        VERBOSE  WARNINGS  XREF
    
    
    This grammar is LALR(1).
    
    Number of Terminals: 105
    Number of Nonterminals: 202
    Number of Productions: 437
    Number of Single Productions: 162
    Number of Items: 1265
    Number of States: 591
    Number of Shift actions: 3482
    Number of Goto actions: 4061
    Number of Shift/Reduce actions: 369
    Number of Goto/Reduce actions: 687
    Number of Reduce actions: 7736
    Number of Shift-Reduce conflicts: 0
    Number of Reduce-Reduce conflicts: 0
    Number of Reductions saved by default: 4913
    Reallocating storage for TIME table, adding 3603 entries
    
    Length of Check table: 16836
    Length of Action table: 16608
    Number of entries in Action Table: 12013
    Percentage of increase: 38.2%
    Highest symbol in Check Table: 307
    Storage Required for Tables: 66888 Bytes, 66K
    Storage Required for Rules: 1308 Bytes
    
    
    Actions in Compressed Tables:
         Number of Shifts: 3482
         Number of Shift/Reduces: 369
         Number of Gotos: 4061
         Number of Goto/Reduces: 687
         Number of Reduces: 2823
         Number of Defaults: 390
    
    Error maps storage:
        Storage required for ACTION_SYMBOLS_BASE map: 1182 Bytes
        Storage required for ACTION_SYMBOLS_RANGE map: 1007 Bytes
        Storage required for NACTION_SYMBOLS_BASE map: 1182 Bytes
        Storage required for NACTION_SYMBOLS_RANGE map: 630 Bytes
        Storage required for SYMBOL_INDEX map: 616 Bytes
        Storage required for STRING_BUFFER map: 4652 Bytes
    
    ***Warning: Base Check vector contains value > 127. 16-bit words used.
    ***Warning: Terminal symbol > 127. 16-bit words used.
    Escaped symbol $eof is an invalid C variable.
    
    Escaped symbol $error is an invalid C variable.
     
    It can be quite different if the output changed since the version 2.30 of lpg. The important part is:
    This grammar is LALR(1).
    This creates in the current directory some java source files and information files.
    java.l Information generated by lpg/jikespg. Enumarate all the states created for the automaton, etc.
    JavaAction.java It contains the method consumeRule(int) of the class org.eclipse.jdt.internal.compiler.parser.Parser that handles all semantic actions dispatches.
    javahdr.java You don't need this file. It is actually empty.
    javadcl.java This files is used to generate the resources files.
    javasym.java This is the contents of the class org.eclipse.jdt.core.compiler.ITerminalSymbols. You need to replace:
    • TokenName$eof with TokenNameEOF
    • TokenName$error with TokenNameERROR
    javadef.java This is the contents of the class org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation.
    javaprs.java You don't need this file. Its contents is already inlined in the Parser class.
  3. Now we need to update the different classes and resource files.
    1. Copy the contents of the JavaAction.java file into the consumeRule(int) method of the org.eclipse.jdt.internal.compiler.parser.Parser class.
    2. The definition of the Parser needs to be updated with two tables from javadcl.java. Those are rhs[] and name[]. The following entries in name[] need to be replaced:
      • $eof with UNEXPECTED_EOF
      • $error with "Invalid Character"
      The previous definition of name[] will guide you.
    3. The class org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation needs to be updated with the content of the file javadef.java. Don't copy the interface name. Simply copy the field declarations. The actual source of this class will guide you.
    4. This is the contents of the class org.eclipse.jdt.internal.compiler.parser.TerminalSymbols. You need to replace:
      • TokenName$eof with TokenNameEOF
      • TokenName$error with TokenNameERROR
    5. The last step is to update the resource files:
      Copy the jdtcore.jar file in d:\temp. Compile this source inside d:\temp. You will have a file UpdateParserFiles.class. Then run the following command-line:
       D:\temp>java -classpath jdtcore.jar;. UpdateParserFiles javadcl.java
       
      Once this done, you will end up with 5 new files inside d:\temp. They are called parser<n>.rsc, with n equals to 1..5. All these files need to be moved to the org\eclipse\jdt\internal\compiler\parser folder. Now you are ready to execute and test the new parser.

NOTE:

Changing the parser is a risky operation if you miss one of the steps above. The resulting parser can be completely unpredictable. It can go from crashing to reporting invalid errors. Be sure that you followed all the steps and that all the files are updated and recompiled before you run it.