Fix moc preprocessor-only mode with input that contains seemingly invalid identifiers

In WebKit we use moc -E to pre-process various files before throwing at
further build creation tools. The pre-processing is used to filter out
code depending in #ifdef'fed features.

The latest addition to the family of pre-processed files is the CSS grammar,
which is written in Bison. It contains rule lines like

   $$ = parser->createFoo()

and when pre-processing this moc stumbles over the dollar sign. Instead
of ignoring un-tokenizable input we should add it to the current token
if we're in preprocessor-only mode, otherwise the $$ gets eaten and we
produce data-loss by printing out less characters than.

Change-Id: Ib32e7c04b38dd2ba3726201e76f27405f7ea6c0d
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>

1 files changed, 5 insertions, 2 deletions

diff --git a/src/tools/moc/preprocessor.cpp b/src/tools/moc/preprocessor.cpp
index fd8813ee88..40bba33d40 100644
--- a/src/tools/moc/preprocessor.cpp
+++ b/src/tools/moc/preprocessor.cpp
@@ -193,9 +193,12 @@ static Symbols tokenize(const QByteArray &input, int lineNum = 1, TokenizeMode m
                 token = keywords[state].ident;
 
             if (token == NOTOKEN) {
-                // an error really
                 ++data;
-                continue;
+                // an error really, but let's ignore this input
+                // to not confuse moc later. However in pre-processor
+                // only mode let's continue.
+                if (!Preprocessor::preprocessOnly)
+                    continue;
             }
 
             ++column;