Allow non-character codes in utf8 strings

Changed the processing of non-character code handling in the UTF8 codec.
Non-character codes are now accepted in QStrings, QUrls and QJson strings.
Unit tests were adapted accordingly.
For more info about non-character codes,
see: http://www.unicode.org/versions/corrigendum9.html

[ChangeLog][QtCore][QUtf8]
UTF-8 now accepts non-character unicode points; these are not replaced
by the replacement character anymore

[ChangeLog][QtCore][QUrl]
QUrl now fully accepts non-character unicode points; they are encoded as
percent characters; they can also be pretty decoded

[ChangeLog][QtCore][QJson]
The Writer and the Parser now fully accept non-character unicode points.

Change-Id: I77cf4f0e6210741eac8082912a0b6118eced4f77
Task-number: QTBUG-33229
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>

1 files changed, 9 insertions, 11 deletions

diff --git a/tests/auto/corelib/codecs/utf8/tst_utf8.cpp b/tests/auto/corelib/codecs/utf8/tst_utf8.cpp
index 99147f3aff..e18f6f73b9 100644
--- a/tests/auto/corelib/codecs/utf8/tst_utf8.cpp
+++ b/tests/auto/corelib/codecs/utf8/tst_utf8.cpp
@@ -233,8 +233,9 @@ void tst_Utf8::nonCharacters_data()
     QTest::addColumn<QByteArray>("utf8");
     QTest::addColumn<QString>("utf16");
 
-    // Unicode has a couple of "non-characters" that one can use internally,
-    // but are not allowed to be used for text interchange.
+    // Unicode has a couple of "non-characters" that one can use internally
+    // These characters may be used for interchange;
+    // see: http://www.unicode.org/versions/corrigendum9.html
     //
     // Those are the last two entries each Unicode Plane (U+FFFE, U+FFFF,
     // U+1FFFE, U+1FFFF, etc.) as well as the entries between U+FDD0 and
@@ -279,20 +280,17 @@ void tst_Utf8::nonCharacters()
     decoder->toUnicode(utf8);
 
     // Only enforce correctness on our UTF-8 decoder
-    // The system's UTF-8 codec is sometimes buggy
-    //  GNU libc's iconv is known to accept U+FFFF and U+FFFE encoded as UTF-8
-    //  OS X's iconv is known to accept those, plus surrogates and codepoints above U+10FFFF
     if (!useLocale)
-        QVERIFY(decoder->hasFailure());
-    else if (!decoder->hasFailure())
-        qWarning("System codec does not report failure when it should. Should report bug upstream.");
+        QVERIFY(!decoder->hasFailure());
+    else if (decoder->hasFailure())
+        qWarning("System codec reports failure when it shouldn't. Should report bug upstream.");
 
     QSharedPointer<QTextEncoder> encoder(codec->makeEncoder());
     encoder->fromUnicode(utf16);
     if (!useLocale)
-        QVERIFY(encoder->hasFailure());
-    else if (!encoder->hasFailure())
-        qWarning("System codec does not report failure when it should. Should report bug upstream.");
+        QVERIFY(!encoder->hasFailure());
+    else if (encoder->hasFailure())
+        qWarning("System codec reports failure when it shouldn't. Should report bug upstream.");
 }
 
 QTEST_MAIN(tst_Utf8)