[MLton] fix for bug in mlyacc

Michael Norrish Michael.Norrish at nicta.com.au
Tue Jan 11 21:26:03 PST 2011


Fix bug in comment-handling in lexer for mlyacc's input language.

In particular, if a comment was successfully lexed, it returned a BOGUS_VALUE
token, which then messed up the higher-level parsing.  This was due to
the use of the idiom

   continue() before YYBEGIN SOMESTATE

making the return value of the continue the return value for the lexer at
that point.

This bug prevented the use of comments in code sections entirely.

It was somewhat masked in the prefix section by the fact that most
supposed comments there were never lexed as such anyway.  This could
be observed by a file that included %% in a comment in the prefix
section.


diff --git a/mlyacc/src/yacc.lex b/mlyacc/src/yacc.lex
--- a/mlyacc/src/yacc.lex
+++ b/mlyacc/src/yacc.lex
@@ -71,11 +71,11 @@
  qualid ={id}".";
  %%
  <INITIAL>"(*"	=> (Add yytext; YYBEGIN COMMENT; commentLevel := 1;
-		    continue() before YYBEGIN INITIAL);
+		    continue(); YYBEGIN INITIAL; continue());
  <A>"(*"		=> (YYBEGIN EMPTYCOMMENT; commentLevel := 1; continue());
  <CODE>"(*"	=> (Add yytext; YYBEGIN COMMENT; commentLevel := 1;
-		    continue() before YYBEGIN CODE);
-<INITIAL>[^%\n]+ => (Add yytext; continue());
+		    continue(); YYBEGIN CODE; continue());
+<INITIAL>[^(%\n]+ => (Add yytext; continue());
  <INITIAL>"%%"	 => (YYBEGIN A; HEADER (concat (rev (!text)),!lineno,!lineno));
  <INITIAL,CODE,COMMENT,F,EMPTYCOMMENT>\n  => (Add yytext; inc lineno; continue());
  <INITIAL>.	 => (Add yytext; continue());



More information about the MLton mailing list