include File / EOF handling
|
View:
New views
1 Messages
—
Rating Filter:
Alert me
|
 |
include File / EOF handling
by Wolfgang Fahl-2
::
Rate this Message:
Some parts of this message have been removed.
Learn more about Nabble's security policy.
Dear javaCC users, javaCC is an excellent parser generator which we have been using since 1998 within our smartGENERATOR Model Driven Development tool. A few days ago we have upgraded to a newer version JavaCC 4 after we had not touched anything for about seven years. Our Macro/Template language has the #include feature and we used to implement this by a modified ASCII_UCodeESC_CharStream as described in https://javacc.dev.java.net/doc/CharStream.html see the Source code of 2002 below. with the new JavaCC version 4 we found the following feature: You can now specify actions/state changes for EOF. It is right now very strict in that it has to look exactly like: <*> TOKEN: { < EOF > { action } : NEW_STATE } which means that EOF is still EOF in every state except that now you can specify what state changes if any or what java code if any to execute on seeing EOF. This should help in writing grammars for processing C/C++ #include files, without going through hoops as in the old versions. We tried to use this feature but did not find a proper way how to do so. The includeStack handling push() and pop() methods where moved into the TokenManager class. Pushing and changing the inputstream works nicely. When EOF is seen in the innermost include file the pop() function should make sure that the parser continues with the inputstream popped. The EOF token should be ignored. How can this be accomplished within the TokenManager? The following code doesn't do it: / ** * we have seen an end of file token */ public void eofHandling() { if (Debug.ALL) { optDebug("eof seen in "+activeFileName+"\n",null); } // if if (!includeStack.empty()) { pop(); } } /** * pop the include file from the include file stack */ public void pop() { String oldImage=input_stream.GetImage(); input_stream=includeStack.pop(); activeFileName=(String)fileNameStack.pop(); if (Debug.ALL) { optDebug("popping "+activeFileName+" with old image: "+oldImage+"\n",null); } // if // set the current character try { curChar=input_stream.readChar(); } catch (IOException ioex){ Debug.logit(ioex.getMessage()); } // go back to include state SwitchTo(INCLUDESTATE); } // pop For the time being we are going to implement a work around where the readChar delegates to the TokenManager which knows whether EOF should be returned or not by the state of the includeStack. The announcement by Aliaksandr Radzivanovich at https://javacc.dev.java.net/servlets/ReadMsg?list=dev&msgNo=554
looks very interesting to me. The JavaCC version at: http://wiki.github.com/aradzie/javaccnglooks very promising to me. Especially the different EOF handling gives me hope. What do think? Would you know another include file handling example? The C- Grammar on the grammars page assumes the C code is already preprocessed. This is not an option in my case. Yours Wolfgang -- BITPlan - smart solutions Wolfgang Fahl Pater-Delp-Str. 1, D-47877 Willich Schiefbahn Tel. +49 2154 811-480, Fax +49 2154 811-481 Web: http://www.bitplan.de BITPlan GmbH, Willich - HRB 6820 Krefeld, Steuer-Nr.: 10258040548, Geschäftsführer: Wolfgang Fahl/* Generated By:JavaCC: Do not edit this line. ASCII_UCodeESC_CharStream.java Version 0.7pre6 */ /** * An implementation of interface CharStream, where the stream is assumed to * contain only ASCII characters (with java-like unicode escape processing). */ /* Copyright (C) 1999-2002 BITPlan GmbH Pater-Delp-Str. 1 47877 Willich http://www.bitplan.com Author: Wolfgang Fahl $Header: /usr/cvsroot/Source/Eclipse/com.bitplan.smartGENERATOR/src/com/bitplan/gen/parser/ASCII_UCodeESC_CharStream.java,v 1.7 2006/09/02 08:19:18 msf Exp $ $Log: ASCII_UCodeESC_CharStream.java,v $ Revision 1.7 2006/09/02 08:19:18 msf m o r e Revision 1.6 2006/09/02 08:15:22 msf E O F h a n d l i n g Revision 1.5 2006/09/02 07:17:11 msf f i l e n a m e Revision 1.4 2006/09/02 07:14:42 msf m o r e i n f o Revision 1.3 2006/09/02 06:50:27 msf s t a c k t r a c e a d d e d Revision 1.2 2006/09/02 06:43:14 msf e r r h a n d l i n g Revision 1.1 2006/03/08 14:20:31 wf new files Revision 2.1 2004/07/27 12:36:46 gb get line changed to get end line, says so in the code itself Revision 2.0 2002/11/18 16:10:09 wf *** empty log message *** Revision 1.1 2002/11/05 12:08:13 gb smartGeneator initial CVS checkin Revision 1.8 2002/08/30 11:08:13 wf Header->Id Revision 1.7 2002/08/27 09:10:52 wf pop at EOF with BeginToken Revision 1.6 2002/08/27 07:16:01 wf *** empty log message *** Revision 1.5 2002/08/26 17:06:30 wf *** empty log message *** */ package com.bitplan.gen.parser; import java.util.*; import java.io.*; import com.bitplan.common.Debug; /** * The <code>ASCII_UCodeESC_CharStream</code> class is a delegation * interface for the ASCII_UCode_ESC_Char_Stream.java - it's used to delegate calls * to the Original one. This is a static version of ASCII_UCodeESC_CharStream. * * It keeps track of the IncludeFileStack and handles EOFs in a transparent manner by popping the inputstream * from the includestack when the end of an IncludeFile has been reached. * * Note: There is only minimal support for a token that crosses the boundary of include files. The "backup" function * isn't supported so only tokens that don't need any decision making will be scanned correctly. The main purpose * is to support SPECIAL-Tokens for the Macro-Generator that have the normal source-code across the boundary * of include-files. * <p> * @author Wolfgang Fahl * @version $Revision: 1.7 $ */ public final class ASCII_UCodeESC_CharStream { public static String RCSID="$Id: ASCII_UCodeESC_CharStream.java,v 1.7 2006/09/02 08:19:18 msf Exp $"; /** * This is the original input stream that calls are delegated to * */ static private UStream inputStream=null; /** * Beginning of Token from previous file on stack * */ static private String OldImage=null; /* Include - File handling */ /** * the currently active Filename * */ public static String activeFileName=""; /** * The stack for the include Files (UStream's) * This stack is kept in sync with the FileNameStack */ private static Stack IncludeStack =new Stack(); /** * The stack for the include Files Names of the UStreams * This stack is kept in sync with the IncludeStack */ private static Stack FileNameStack=new Stack(); /** * set the active filename * @param aFileName the Filename of the file that is currently scanned/parsed * */ public static void setFileName(String aFileName) { activeFileName=aFileName; } /** * the default include path */ private static String includePath="."; /** * set the includePath as asked for by m2java.jjt */ public static void setIncludePath(String aPath) { includePath=aPath; } /** * tell if there are still more files to be parsed * @return true if there is still one or more files to be scanned on the include-stack * */ public static boolean more() { if (Debug.ALL) Debug.logit("ASCII_UCodeESC_CharStream was asked for more "+!IncludeStack.isEmpty()); return !(IncludeStack.isEmpty()); } /** * push the current file on the include stack and switch to a new one * @param pFileName - name of the file to be included next * */ public static void push(String pFileName) { IncludeStack.push(inputStream); if (Debug.CORE) Debug.logit(" pushing: "+activeFileName+" on include stack"); FileNameStack.push(activeFileName); String FileName=includePath+java.io.File.separator+pFileName; StringTokenizer PathSplit = new StringTokenizer(includePath,File.pathSeparator); while (PathSplit.hasMoreTokens()) { String TestPath=PathSplit.nextToken(); // System.out.println(TestPath); String FilePath=TestPath+java.io.File.separator+pFileName; File TestFile=new File(FilePath); // System.out.println(FilePath); if (TestFile.exists()) { FileName=FilePath; break; } } activeFileName=FileName; try { if (Debug.CORE) Debug.log(" IncludeFile: "+FileName+" ..."); UStream newinput=new UStream(new FileInputStream(FileName),1,1); inputStream=newinput; } catch (Exception e) { optDebug("Exception in ASCII_UCodeEsc_CharStream.push",e); } } /** * get the active Filename */ public static String getEndFilename() { return activeFileName; } /** * optionally show a debug message * @param t */ public static void optDebug(String msg,Throwable t) { Debug.logit(msg); Debug.logit("at "+getEndFilename()+" line "+inputStream.getEndLine()+" col "+inputStream.getEndColumn()); // if there was a Throwable show it's details if (t!=null) { Debug.logit("Message: "+t.getMessage()); Debug.logit("Stacktrace: "+com.bitplan.common.ExceptionStack.getStackTraceText(t)); } // if } // optDebug /** * pop the include file from the include file stack */ public static void pop() { OldImage=GetImage(); // remember last Image inputStream=(UStream)IncludeStack.pop(); activeFileName=(String)FileNameStack.pop(); if (Debug.ALL) { optDebug("popping "+activeFileName+" with old image: "+OldImage+"\n",null); } // if } // pop /** * read a a character from the input * @return - the character read * @throws java.io.IOException - if reading fails */ public static char readChar() throws java.io.IOException { char result='?'; try { result =inputStream.readChar(); //if (Debug.ALL) // Debug.logit("read <"+result+">"); } catch (java.io.IOException e) { // EOF found in inputstream if (Debug.CORE) { optDebug("exception with readChar more="+more(),e); } // if if (more()) { // Just an include file don't propagate EOF pop(); // Just switch to new (old) file result=inputStream.readChar(); // and read the next character } else throw new java.io.IOException(e.getMessage()); // signal EOF to macroTokenManager } catch (Throwable t) { // some other problem if (Debug.CORE) { optDebug("throwable with readChar more="+more(),t); } } return result; } // readChar /** * BeginToken * @return * @throws java.io.IOException */ static public final char BeginToken() throws java.io.IOException{ char result='?'; try { OldImage=null; // new Token - forget OldImage result=inputStream.BeginToken(); } catch (java.io.IOException e) { // EOF found in inputstream if (Debug.CORE) { if (e.getMessage()==null) optDebug("Assuming EOF Found more="+more(),null); else optDebug("EOF expected - is the io exception correct? more="+more(),e); } // if if (more()) { // Just an include file don't propagate EOF pop(); // Just switch to new (old) file result=inputStream.BeginToken(); // and read the next character } else { throw new java.io.IOException(e.getMessage()); // signal EOF to macroTokenManager } // if } // try return result; } // BeginToken /** * Get the current image * @return */ static public final String GetImage() { String result=inputStream.GetImage(); if (OldImage!=null) { // leftover from previous file? result=OldImage+result; // add in front of image OldImage=null; // no leftovers now } return result; } // GetImage static public final char[] GetSuffix(int len) { return inputStream.GetSuffix(len); } static public void Done() { inputStream.Done(); } static public final int getLine() { return inputStream.getEndLine(); } static public final int getEndColumn() { return inputStream.getEndColumn(); } static public final int getEndLine() { return inputStream.getEndLine(); } static public final int getBeginColumn() { return inputStream.getBeginColumn(); } static public final int getBeginLine() { return inputStream.getBeginLine(); } static public final void backup(int amount) { inputStream.backup(amount); } /** * delegate constructor */ public ASCII_UCodeESC_CharStream(java.io.InputStream dstream,int startline, int startcolumn) { inputStream=new UStream(dstream,startline,startcolumn); } /** * delegate Reinit */ public void ReInit(java.io.InputStream dstream,int startline, int startcolumn) { if (inputStream!=null) inputStream.ReInit(dstream,startline,startcolumn); } /** * delegate constructor */ public ASCII_UCodeESC_CharStream(java.io.Reader dstream,int startline, int startcolumn) { inputStream=new UStream(dstream,startline,startcolumn); } /** * delegate Reinit */ public static void ReInit(java.io.Reader dstream, int startline, int startcolumn, int buffersize) { if (inputStream!=null) inputStream.ReInit(dstream,startline,startcolumn,buffersize); } /** * delegate Reinit */ public void ReInit(java.io.Reader dstream,int startline, int startcolumn) { if (inputStream!=null) inputStream.ReInit(dstream,startline,startcolumn); } }
From: Aliaksandr Radzivanovich <aradzivanovich@...> Date: Fri, 16 Oct 2009 18:32:54 +0300 Content-Type: multipart/alternative; boundary=001485f6db1640ccad04760f1d2e Subject: [ANN] javacc with bug fixes available Hello javacc developers and users. I am involved in a project that relies on javacc heavily. In my opinion javacc in one of the best tools but still it has some defects. I've personally filed bug report 229 that is a show stopper for my project, but this bug has not been fixed for a while. So I decided to bite the bullet and fix it by myself. If fact, I've fixed not only that bug, but some more issues that were in my to do list. I did this at the cost of breaking backward compatibility of the generated code. But I believe that benefit of having cleaner and better code outweights compatibility breakage. After all, porting to the new version should be trivial, since grammar and tools have not changed. If you would like to take a look at it, you fill find its home on github: http://github.com/aradzie/javaccng in a repository named javaccng. The summary of changes is on the default wiki page: http://wiki.github.com/aradzie/javaccng. The new javacc serves well my needs, but I would be glad if you find it useful as well and integrate my changes to the main javacc development trunk. May be it could become a foundation for the new version. Please ask me for any assistance when you need it. Responses
|
| Free embeddable forum powered by Nabble | Forum Help |
