[Parser] Update CachedTokens while parsing ObjectiveC template argument list

Consider the following ObjC++ snippet:

--
@protocol PA;
@protocol PB;

@class NSArray<ObjectType>;
typedef int some_t;

id<PA> FA(NSArray<id<PB>> *h, some_t group);
--

This would hit an assertion in the parser after generating an annotation token
while trying to update the token cache:

Assertion failed: (CachedTokens[CachedLexPos-1].getLastLoc() == Tok.getAnnotationEndLoc() && "The annotation should be until the most recent cached token")
...
7 clang::Preprocessor::AnnotatePreviousCachedTokens(clang::Token const&) + 494
8 clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec(bool, bool, clang::CXXScopeSpec&, bool) + 1163
9 clang::Parser::TryAnnotateTypeOrScopeToken(bool, bool) + 361
10 clang::Parser::isCXXDeclarationSpecifier(clang::Parser::TPResult, bool*) + 598
...

The cached preprocessor token in this case is:

greatergreater '>>' Loc=<testcase.mm:7:24>

while the annotation ("NSArray<id<PB>>") ends at "testcase.mm:7:25", hence the
assertion.

Properly update the CachedTokens during template parsing to contain
two greater tokens instead of a greatergreater.

Differential Revision: http://reviews.llvm.org/D15173

rdar://problem/23494277

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@259311 91177308-0d34-0410-b5e6-96231b3b80d8

1 files changed, 11 insertions, 0 deletions

diff --git a/include/clang/Lex/Preprocessor.h b/include/clang/Lex/Preprocessor.h
index f6154b6a49..5ffa48277f 100644
--- a/include/clang/Lex/Preprocessor.h
+++ b/include/clang/Lex/Preprocessor.h
@@ -1185,6 +1185,17 @@ public:
     return CachedTokens[CachedLexPos-1].getLastLoc();
   }
 
+  /// \brief Whether \p Tok is the most recent token (`CachedLexPos - 1`) in
+  /// CachedTokens.
+  bool IsPreviousCachedToken(const Token &Tok) const;
+
+  /// \brief Replace token in `CachedLexPos - 1` in CachedTokens by the tokens
+  /// in \p NewToks.
+  ///
+  /// Useful when a token needs to be split in smaller ones and CachedTokens
+  /// most recent token must to be updated to reflect that.
+  void ReplacePreviousCachedToken(ArrayRef<Token> NewToks);
+
   /// \brief Replace the last token with an annotation token.
   ///
   /// Like AnnotateCachedTokens(), this routine replaces an