Fix the fix to correctly handle comments in multi-line assignments

The previous fix where the Grammar comment style was changed to
remove the newlines was incorrect, because if you have

 foo=1#comment
 bar=2

after the Grammar comment ignoring, it would transform into

 foo=1bar=2

which will clearly fail to parse, so the new line has to stay.

But we would still have the following case which would fail:
 foo=a \
 # comment
 b

Apparently qmake things that's the equivalent of
foo=a b

but the grammar parses it as
 foo=a \
 \n (newline)
 b

Thus the parsing fails because there's a newline and then some
weird 'b' token which the grammar does not expect.

The best fix I found is to preprocess the source, to remove
completely commented out lines.

So:
 foo=a \
 # comment
 b

gets transformed into

 foo=a \
 b

Change-Id: I2487a0dbf94a6ad4d917d0a0ce05247341e9b7da
Reviewed-by: Tobias Hunger <tobias.hunger@qt.io>

3 files changed, 42 insertions, 7 deletions

diff --git a/util/cmake/pro2cmake.py b/util/cmake/pro2cmake.py
index ba77e0e51f..0036c658d6 100755
--- a/util/cmake/pro2cmake.py
+++ b/util/cmake/pro2cmake.py
@@ -155,6 +155,27 @@ def fixup_linecontinuation(contents: str) -> str:
     return contents
 
 
+def fixup_comments(contents: str) -> str:
+    # Get rid of completely commented out lines.
+    # So any line which starts with a '#' char and ends with a new line
+    # will be replaced by a single new line.
+    #
+    # This is needed because qmake syntax is weird. In a multi line
+    # assignment (separated by backslashes and newlines aka
+    # # \\\n ), if any of the lines are completely commented out, in
+    # principle the assignment should fail.
+    #
+    # It should fail because you would have a new line separating
+    # the previous value from the next value, and the next value would
+    # not be interpreted as a value, but as a new token / operation.
+    # qmake is lenient though, and accepts that, so we need to take
+    # care of it as well, as if the commented line didn't exist in the
+    # first place.
+
+    contents = re.sub(r'\n#[^\n]*?\n', '\n', contents, re.DOTALL)
+    return contents
+
+
 def spaces(indent: int) -> str:
     return '    ' * indent
 
@@ -782,12 +803,7 @@ class QmakeParser:
                 expr.setDebug()
 
         Grammar = StatementGroup('statements')
-
-        # Ignore comment lines, including the final line break,
-        # otherwise parsing fails when looking at multi line assignments
-        # with comments in between.
-        Comment = pp.Regex(r"#.*\n").setName("qmake style comment")
-        Grammar.ignore(Comment())
+        Grammar.ignore(pp.pythonStyleComment())
 
         return Grammar
 
@@ -799,6 +815,7 @@ class QmakeParser:
 
             old_contents = contents
             contents = fixup_linecontinuation(contents)
+            contents = fixup_comments(contents)
 
             if old_contents != contents:
                 print('Warning: Fixed line continuation in .pro-file!\n'
diff --git a/util/cmake/tests/data/lc_with_comment.pro b/util/cmake/tests/data/lc_with_comment.pro
index c087dadacc..176913dfc8 100644
--- a/util/cmake/tests/data/lc_with_comment.pro
+++ b/util/cmake/tests/data/lc_with_comment.pro
@@ -2,3 +2,21 @@ SUBDIRS = \
 #    dds \
     tga \
     wbmp
+
+MYVAR = foo # comment
+MYVAR = foo2# comment
+MYVAR = foo3# comment #
+
+MYVAR = foo4# comment #
+
+##
+#
+#
+##
+
+ #
+ #
+#
+  # #
+
+MYVAR = foo5# comment # #
diff --git a/util/cmake/tests/test_parsing.py b/util/cmake/tests/test_parsing.py
index 4b6f48b931..c8feeb1811 100755
--- a/util/cmake/tests/test_parsing.py
+++ b/util/cmake/tests/test_parsing.py
@@ -308,4 +308,4 @@ def test_realworld_lc():
 
 def test_realworld_lc_with_comment_in_between():
     result = parse_file(_tests_path + '/data/lc_with_comment.pro')
-    assert len(result) == 1
+    assert len(result) == 6