Add QTextMarkdownWriter, QTextEdit::markdown property etc.

A QTextDocument can now be written out in Markdown format.

- Add the QTextMarkdownWriter as a private class for now
- Add QTextDocument::toMarkdown()
- QTextDocumentWriter uses QTextMarkdownWriter if setFormat("markdown")
  is called or if the file suffix is .md or .mkd
- Add QTextEdit::toMarkdown() and the markdown property

[ChangeLog][QtGui][Text] Markdown (CommonMark or GitHub dialect) is now
a supported format for reading into and writing from QTextDocument.

Change-Id: I663a77017fac7ae1b3f9a400f5cd357bb40750af
Reviewed-by: Gatis Paeglis <gatis.paeglis@qt.io>

7 files changed, 538 insertions, 0 deletions

diff --git a/tests/auto/gui/text/qtextmarkdownwriter/BLACKLIST b/tests/auto/gui/text/qtextmarkdownwriter/BLACKLIST
new file mode 100644
index 0000000000..fc9e5a9efe
--- /dev/null
+++ b/tests/auto/gui/text/qtextmarkdownwriter/BLACKLIST
@@ -0,0 +1,3 @@
+[rewriteDocument]
+winrt # QTBUG-54623
+
diff --git a/tests/auto/gui/text/qtextmarkdownwriter/data/example.md b/tests/auto/gui/text/qtextmarkdownwriter/data/example.md
new file mode 100644
index 0000000000..3c63f209a2
--- /dev/null
+++ b/tests/auto/gui/text/qtextmarkdownwriter/data/example.md
@@ -0,0 +1,95 @@
+# QTextEdit
+
+The QTextEdit widget is an advanced editor that supports formatted rich text.
+It can be used to display HTML and other rich document formats. Internally,
+QTextEdit uses the QTextDocument class to describe both the high-level
+structure of each document and the low-level formatting of paragraphs.
+
+If you are viewing this document in the textedit example, you can edit this
+document to explore Qt's rich text editing features. We have included some
+comments in each of the following sections to encourage you to experiment.
+
+## Font and Paragraph Styles
+
+QTextEdit supports **bold**, *italic*, and ~~strikethrough~~ font styles, and can
+display multicolored text. Font families such as Times New Roman and `Courier`
+can also be used directly.  *If you place the cursor in a region of styled text,
+the controls in the tool bars will change to reflect the current style.*
+
+Paragraphs can be formatted so that the text is left-aligned, right-aligned,
+centered, or fully justified.
+
+*Try changing the alignment of some text and resize the editor to see how the
+text layout changes.*
+
+## Lists
+
+Different kinds of lists can be included in rich text documents. Standard
+bullet lists can be nested, using different symbols for each level of the list:
+
+* Disc symbols are typically used for top-level list items.
+  - Circle symbols can be used to distinguish between items in lower-level
+    lists.
+    + Square symbols provide a reasonable alternative to discs and circles.
+
+Ordered lists can be created that can be used for tables of contents. Different
+characters can be used to enumerate items, and we can use both Roman and Arabic
+numerals in the same list structure:
+
+1. Introduction
+2. Qt Tools
+    1) Qt Assistant
+    2) Qt Designer
+        1. Form Editor
+        2. Component Architecture
+    3) Qt Linguist
+
+The list will automatically be renumbered if you add or remove items. *Try
+adding new sections to the above list or removing existing item to see the
+numbers change.*
+
+## Images
+
+Inline images are treated like ordinary ranges of characters in the text
+editor, so they flow with the surrounding text. Images can also be selected in
+the same way as text, making it easy to cut, copy, and paste them.
+
+![image](images/logo32.png) *Try to select this image by clicking and dragging
+over it with the mouse, or use the text cursor to select it by holding down
+Shift and using the arrow keys. You can then cut or copy it, and paste it into
+different parts of this document.*
+
+## Tables
+
+QTextEdit can arrange and format tables, supporting features such as row and
+column spans, text formatting within cells, and size constraints for columns.
+
+
+|             |Development Tools                   |Programming Techniques     |Graphical User Interfaces|
+|-------------|------------------------------------|---------------------------|-------------------------|
+|9:00 - 11:00 |Introduction to Qt                                                                      |||
+|11:00 - 13:00|Using qmake                         |Object-oriented Programming|Layouts in Qt            |
+|13:00 - 15:00|Qt Designer Tutorial                |Extreme Programming        |Writing Custom Styles    |
+|15:00 - 17:00|Qt Linguist and Internationalization|                           |                         |
+
+*Try adding text to the cells in the table and experiment with the alignment of
+the paragraphs.*
+
+## Hyperlinks
+
+QTextEdit is designed to support hyperlinks between documents, and this feature
+is used extensively in 
+[Qt Assistant](http://doc.qt.io/qt-5/qtassistant-index.html). Hyperlinks are
+automatically created when an HTML file is imported into an editor. Since the
+rich text framework supports hyperlinks natively, they can also be created
+programatically.
+
+## Undo and Redo
+
+Full support for undo and redo operations is built into QTextEdit and the
+underlying rich text framework. Operations on a document can be packaged
+together to make editing a more comfortable experience for the user.
+
+*Try making changes to this document and press `Ctrl+Z` to undo them. You can
+always recover the original contents of the document.*
+
diff --git a/tests/auto/gui/text/qtextmarkdownwriter/qtextmarkdownwriter.pro b/tests/auto/gui/text/qtextmarkdownwriter/qtextmarkdownwriter.pro
new file mode 100644
index 0000000000..04cf7ef5dd
--- /dev/null
+++ b/tests/auto/gui/text/qtextmarkdownwriter/qtextmarkdownwriter.pro
@@ -0,0 +1,7 @@
+CONFIG += testcase
+TARGET = tst_qtextmarkdownwriter
+QT += core-private gui-private testlib
+SOURCES += tst_qtextmarkdownwriter.cpp
+TESTDATA += data/example.md
+
+DEFINES += SRCDIR=\\\"$$PWD\\\"
diff --git a/tests/auto/gui/text/qtextmarkdownwriter/tst_qtextmarkdownwriter.cpp b/tests/auto/gui/text/qtextmarkdownwriter/tst_qtextmarkdownwriter.cpp
new file mode 100644
index 0000000000..bf7c9708de
--- /dev/null
+++ b/tests/auto/gui/text/qtextmarkdownwriter/tst_qtextmarkdownwriter.cpp
@@ -0,0 +1,360 @@
+/****************************************************************************
+**
+** Copyright (C) 2019 The Qt Company Ltd.
+** Contact: https://www.qt.io/licensing/
+**
+** This file is part of the test suite of the Qt Toolkit.
+**
+** $QT_BEGIN_LICENSE:GPL-EXCEPT$
+** Commercial License Usage
+** Licensees holding valid commercial Qt licenses may use this file in
+** accordance with the commercial license agreement provided with the
+** Software or, alternatively, in accordance with the terms contained in
+** a written agreement between you and The Qt Company. For licensing terms
+** and conditions see https://www.qt.io/terms-conditions. For further
+** information use the contact form at https://www.qt.io/contact-us.
+**
+** GNU General Public License Usage
+** Alternatively, this file may be used under the terms of the GNU
+** General Public License version 3 as published by the Free Software
+** Foundation with exceptions as appearing in the file LICENSE.GPL3-EXCEPT
+** included in the packaging of this file. Please review the following
+** information to ensure the GNU General Public License requirements will
+** be met: https://www.gnu.org/licenses/gpl-3.0.html.
+**
+** $QT_END_LICENSE$
+**
+****************************************************************************/
+
+#include <QtTest/QtTest>
+#include <QTextDocument>
+#include <QTextCursor>
+#include <QTextBlock>
+#include <QTextList>
+#include <QTextTable>
+#include <QBuffer>
+#include <QDebug>
+
+#include <private/qtextmarkdownwriter_p.h>
+
+// #define DEBUG_WRITE_OUTPUT
+
+class tst_QTextMarkdownWriter : public QObject
+{
+    Q_OBJECT
+public slots:
+    void init();
+    void cleanup();
+
+private slots:
+    void testWriteParagraph_data();
+    void testWriteParagraph();
+    void testWriteList();
+    void testWriteNestedBulletLists();
+    void testWriteNestedNumericLists();
+    void testWriteTable();
+    void rewriteDocument();
+    void fromHtml_data();
+    void fromHtml();
+
+private:
+    QString documentToUnixMarkdown();
+
+private:
+    QTextDocument *document;
+};
+
+void tst_QTextMarkdownWriter::init()
+{
+    document = new QTextDocument();
+}
+
+void tst_QTextMarkdownWriter::cleanup()
+{
+    delete document;
+}
+
+void tst_QTextMarkdownWriter::testWriteParagraph_data()
+{
+    QTest::addColumn<QString>("input");
+    QTest::addColumn<QString>("output");
+
+    QTest::newRow("empty") << "" <<
+        "";
+    QTest::newRow("spaces") << "foobar   word" <<
+        "foobar   word\n\n";
+    QTest::newRow("starting spaces") << "  starting spaces" <<
+        "  starting spaces\n\n";
+    QTest::newRow("trailing spaces") << "trailing spaces  " <<
+        "trailing spaces  \n\n";
+    QTest::newRow("tab") << "word\ttab x" <<
+        "word\ttab x\n\n";
+    QTest::newRow("tab2") << "word\t\ttab\tx" <<
+        "word\t\ttab\tx\n\n";
+    QTest::newRow("misc") << "foobar   word\ttab x" <<
+        "foobar   word\ttab x\n\n";
+    QTest::newRow("misc2") << "\t     \tFoo" <<
+        "\t     \tFoo\n\n";
+}
+
+void tst_QTextMarkdownWriter::testWriteParagraph()
+{
+    QFETCH(QString, input);
+    QFETCH(QString, output);
+
+    QTextCursor cursor(document);
+    cursor.insertText(input);
+
+    QCOMPARE(documentToUnixMarkdown(), output);
+}
+
+void tst_QTextMarkdownWriter::testWriteList()
+{
+    QTextCursor cursor(document);
+    QTextList *list = cursor.createList(QTextListFormat::ListDisc);
+    cursor.insertText("ListItem 1");
+    list->add(cursor.block());
+    cursor.insertBlock();
+    cursor.insertText("ListItem 2");
+    list->add(cursor.block());
+
+    QCOMPARE(documentToUnixMarkdown(), QString::fromLatin1(
+        "- ListItem 1\n- ListItem 2\n"));
+}
+
+void tst_QTextMarkdownWriter::testWriteNestedBulletLists()
+{
+    QTextCursor cursor(document);
+
+    QTextList *list1 = cursor.createList(QTextListFormat::ListDisc);
+    cursor.insertText("ListItem 1");
+    list1->add(cursor.block());
+
+    QTextListFormat fmt2;
+    fmt2.setStyle(QTextListFormat::ListCircle);
+    fmt2.setIndent(2);
+    QTextList *list2 = cursor.insertList(fmt2);
+    cursor.insertText("ListItem 2");
+
+    QTextListFormat fmt3;
+    fmt3.setStyle(QTextListFormat::ListSquare);
+    fmt3.setIndent(3);
+    cursor.insertList(fmt3);
+    cursor.insertText("ListItem 3");
+
+    cursor.insertBlock();
+    cursor.insertText("ListItem 4");
+    list1->add(cursor.block());
+
+    cursor.insertBlock();
+    cursor.insertText("ListItem 5");
+    list2->add(cursor.block());
+
+    QCOMPARE(documentToUnixMarkdown(), QString::fromLatin1(
+                 "- ListItem 1\n  * ListItem 2\n    + ListItem 3\n- ListItem 4\n  * ListItem 5\n"));
+}
+
+void tst_QTextMarkdownWriter::testWriteNestedNumericLists()
+{
+    QTextCursor cursor(document);
+
+    QTextList *list1 = cursor.createList(QTextListFormat::ListDecimal);
+    cursor.insertText("ListItem 1");
+    list1->add(cursor.block());
+
+    QTextListFormat fmt2;
+    fmt2.setStyle(QTextListFormat::ListLowerAlpha);
+    fmt2.setNumberSuffix(QLatin1String(")"));
+    fmt2.setIndent(2);
+    QTextList *list2 = cursor.insertList(fmt2);
+    cursor.insertText("ListItem 2");
+
+    QTextListFormat fmt3;
+    fmt3.setStyle(QTextListFormat::ListDecimal);
+    fmt3.setIndent(3);
+    cursor.insertList(fmt3);
+    cursor.insertText("ListItem 3");
+
+    cursor.insertBlock();
+    cursor.insertText("ListItem 4");
+    list1->add(cursor.block());
+
+    cursor.insertBlock();
+    cursor.insertText("ListItem 5");
+    list2->add(cursor.block());
+
+    // There's no QTextList API to set the starting number so we hard-coded all lists to start at 1 (QTBUG-65384)
+    QCOMPARE(documentToUnixMarkdown(), QString::fromLatin1(
+                 "1 ListItem 1\n    1) ListItem 2\n        1 ListItem 3\n2 ListItem 4\n    2) ListItem 5\n"));
+}
+
+void tst_QTextMarkdownWriter::testWriteTable()
+{
+    QTextCursor cursor(document);
+    QTextTable * table = cursor.insertTable(4, 3);
+    cursor = table->cellAt(0, 0).firstCursorPosition();
+    // valid Markdown tables need headers, but QTextTable doesn't make that distinction
+    // so QTextMarkdownWriter assumes the first row of any table is a header
+    cursor.insertText("one");
+    cursor.movePosition(QTextCursor::NextCell);
+    cursor.insertText("two");
+    cursor.movePosition(QTextCursor::NextCell);
+    cursor.insertText("three");
+    cursor.movePosition(QTextCursor::NextCell);
+
+    cursor.insertText("alice");
+    cursor.movePosition(QTextCursor::NextCell);
+    cursor.insertText("bob");
+    cursor.movePosition(QTextCursor::NextCell);
+    cursor.insertText("carl");
+    cursor.movePosition(QTextCursor::NextCell);
+
+    cursor.insertText("dennis");
+    cursor.movePosition(QTextCursor::NextCell);
+    cursor.insertText("eric");
+    cursor.movePosition(QTextCursor::NextCell);
+    cursor.insertText("fiona");
+    cursor.movePosition(QTextCursor::NextCell);
+
+    cursor.insertText("gina");
+    /*
+        |one   |two |three|
+        |------|----|-----|
+        |alice |bob |carl |
+        |dennis|eric|fiona|
+        |gina  |    |     |
+    */
+
+    QString md = documentToUnixMarkdown();
+
+#ifdef DEBUG_WRITE_OUTPUT
+    {
+        QFile out("/tmp/table.md");
+        out.open(QFile::WriteOnly);
+        out.write(md.toUtf8());
+        out.close();
+    }
+#endif
+
+    QString expected = QString::fromLatin1(
+        "\n|one   |two |three|\n|------|----|-----|\n|alice |bob |carl |\n|dennis|eric|fiona|\n|gina  |    |     |\n\n");
+    QCOMPARE(md, expected);
+
+    // create table with merged cells
+    document->clear();
+    cursor = QTextCursor(document);
+    table = cursor.insertTable(3, 3);
+    table->mergeCells(0, 0, 1, 2);
+    table->mergeCells(1, 1, 1, 2);
+    cursor = table->cellAt(0, 0).firstCursorPosition();
+    cursor.insertText("a");
+    cursor.movePosition(QTextCursor::NextCell);
+    cursor.insertText("b");
+    cursor.movePosition(QTextCursor::NextCell);
+    cursor.insertText("c");
+    cursor.movePosition(QTextCursor::NextCell);
+    cursor.insertText("d");
+    cursor.movePosition(QTextCursor::NextCell);
+    cursor.insertText("e");
+    cursor.movePosition(QTextCursor::NextCell);
+    cursor.insertText("f");
+    /*
+      +---+-+
+      |a  |b|
+      +---+-+
+      |c|  d|
+      +-+-+-+
+      |e|f| |
+      +-+-+-+
+
+      generates
+
+      |a ||b|
+      |-|-|-|
+      |c|d ||
+      |e|f| |
+
+    */
+
+    md = documentToUnixMarkdown();
+
+#ifdef DEBUG_WRITE_OUTPUT
+    {
+        QFile out("/tmp/table-merged-cells.md");
+        out.open(QFile::WriteOnly);
+        out.write(md.toUtf8());
+        out.close();
+    }
+#endif
+
+    QCOMPARE(md, QString::fromLatin1("\n|a ||b|\n|-|-|-|\n|c|d ||\n|e|f| |\n\n"));
+}
+
+void tst_QTextMarkdownWriter::rewriteDocument()
+{
+    QTextDocument doc;
+    QFile f(QFINDTESTDATA("data/example.md"));
+    QVERIFY(f.open(QFile::ReadOnly | QIODevice::Text));
+    QString orig = QString::fromUtf8(f.readAll());
+    f.close();
+    doc.setMarkdown(orig);
+    QString md = doc.toMarkdown();
+
+#ifdef DEBUG_WRITE_OUTPUT
+    QFile out("/tmp/rewrite.md");
+    out.open(QFile::WriteOnly);
+    out.write(md.toUtf8());
+    out.close();
+#endif
+
+    QCOMPARE(md, orig);
+}
+
+void tst_QTextMarkdownWriter::fromHtml_data()
+{
+    QTest::addColumn<QString>("input");
+    QTest::addColumn<QString>("output");
+
+    QTest::newRow("long URL") <<
+        "<span style=\"font-style:italic;\">https://www.example.com/dir/subdir/subsubdir/subsubsubdir/subsubsubsubdir/subsubsubsubsubdir/</span>" <<
+        "*https://www.example.com/dir/subdir/subsubdir/subsubsubdir/subsubsubsubdir/subsubsubsubsubdir/*\n\n";
+    QTest::newRow("non-emphasis inline asterisk") << "3 * 4" << "3 * 4\n\n";
+    QTest::newRow("arithmetic") << "(2 * a * x + b)^2 = b^2 - 4 * a * c" << "(2 * a * x + b)^2 = b^2 - 4 * a * c\n\n";
+    QTest::newRow("escaped asterisk after newline") <<
+        "The first sentence of this paragraph holds 80 characters, then there's a star. * This is wrapped, but is <em>not</em> a bullet point." <<
+        "The first sentence of this paragraph holds 80 characters, then there's a star.\n\\* This is wrapped, but is *not* a bullet point.\n\n";
+    QTest::newRow("escaped plus after newline") <<
+        "The first sentence of this paragraph holds 80 characters, then there's a plus. + This is wrapped, but is <em>not</em> a bullet point." <<
+        "The first sentence of this paragraph holds 80 characters, then there's a plus.\n\\+ This is wrapped, but is *not* a bullet point.\n\n";
+    QTest::newRow("escaped hyphen after newline") <<
+        "The first sentence of this paragraph holds 80 characters, then there's a minus. - This is wrapped, but is <em>not</em> a bullet point." <<
+        "The first sentence of this paragraph holds 80 characters, then there's a minus.\n\\- This is wrapped, but is *not* a bullet point.\n\n";
+    // TODO
+//    QTest::newRow("escaped number and paren after double newline") <<
+//        "<p>(The first sentence of this paragraph is a line, the next paragraph has a number</p>13) but that's not part of an ordered list" <<
+//        "(The first sentence of this paragraph is a line, the next paragraph has a number\n\n13\\) but that's not part of an ordered list\n\n";
+//    QTest::newRow("preformats with embedded backticks") <<
+//        "<pre>none `one` ``two``</pre><pre>```three``` ````four````</pre>plain" <<
+//        "``` none `one` ``two`` ```\n\n````` ```three``` ````four```` `````\n\nplain\n\n";
+}
+
+void tst_QTextMarkdownWriter::fromHtml()
+{
+    QFETCH(QString, input);
+    QFETCH(QString, output);
+
+    document->setHtml(input);
+    QCOMPARE(documentToUnixMarkdown(), output);
+}
+
+QString tst_QTextMarkdownWriter::documentToUnixMarkdown()
+{
+    QString ret;
+    QTextStream ts(&ret, QIODevice::WriteOnly);
+    QTextMarkdownWriter writer(ts, QTextDocument::MarkdownDialectGitHub);
+    writer.writeAll(*document);
+    return ret;
+}
+
+QTEST_MAIN(tst_QTextMarkdownWriter)
+#include "tst_qtextmarkdownwriter.moc"
diff --git a/tests/auto/gui/text/text.pro b/tests/auto/gui/text/text.pro
index 6b033fb506..a98debe35c 100644
--- a/tests/auto/gui/text/text.pro
+++ b/tests/auto/gui/text/text.pro
@@ -28,12 +28,15 @@ SUBDIRS=\
 
 win32:SUBDIRS -= qtextpiecetable
 
+qtConfig(textmarkdownwriter): SUBDIRS += qtextmarkdownwriter
+
 !qtConfig(private_tests): SUBDIRS -= \
            qfontcache \
            qcssparser \
            qtextlayout \
            qtextpiecetable \
            qzip \
+           qtextmarkdownwriter \
            qtextodfwriter
 
 !qtHaveModule(xml): SUBDIRS -= \
diff --git a/tests/manual/markdown/html2md.cpp b/tests/manual/markdown/html2md.cpp
new file mode 100644
index 0000000000..19d6ff06af
--- /dev/null
+++ b/tests/manual/markdown/html2md.cpp
@@ -0,0 +1,64 @@
+/****************************************************************************
+ **
+ ** Copyright (C) 2019 The Qt Company Ltd.
+ ** Contact: https://www.qt.io/licensing/
+ **
+ ** This file is part of the test suite of the Qt Toolkit.
+ **
+ ** $QT_BEGIN_LICENSE:GPL-EXCEPT$
+ ** Commercial License Usage
+ ** Licensees holding valid commercial Qt licenses may use this file in
+ ** accordance with the commercial license agreement provided with the
+ ** Software or, alternatively, in accordance with the terms contained in
+ ** a written agreement between you and The Qt Company. For licensing terms
+ ** and conditions see https://www.qt.io/terms-conditions. For further
+ ** information use the contact form at https://www.qt.io/contact-us.
+ **
+ ** GNU General Public License Usage
+ ** Alternatively, this file may be used under the terms of the GNU
+ ** General Public License version 3 as published by the Free Software
+ ** Foundation with exceptions as appearing in the file LICENSE.GPL3-EXCEPT
+ ** included in the packaging of this file. Please review the following
+ ** information to ensure the GNU General Public License requirements will
+ ** be met: https://www.gnu.org/licenses/gpl-3.0.html.
+ **
+ ** $QT_END_LICENSE$
+ **
+ ****************************************************************************/
+
+#include <QCommandLineParser>
+#include <QDebug>
+#include <QFile>
+#include <QGuiApplication>
+#include <QTextDocument>
+
+int main(int argc, char **argv)
+{
+    QGuiApplication app(argc, argv);
+    QGuiApplication::setApplicationVersion(QT_VERSION_STR);
+    QCommandLineParser parser;
+    parser.setApplicationDescription("Converts the Qt-supported subset of HTML to Markdown.");
+    parser.addHelpOption();
+    parser.addVersionOption();
+    parser.addPositionalArgument(QGuiApplication::translate("main", "input"),
+                                 QGuiApplication::translate("main", "input file"));
+    parser.addPositionalArgument(QGuiApplication::translate("main", "output"),
+                                 QGuiApplication::translate("main", "output file"));
+    parser.process(app);
+    if (parser.positionalArguments().count() != 2)
+        parser.showHelp(1);
+
+    QFile inFile(parser.positionalArguments().first());
+    if (!inFile.open(QIODevice::ReadOnly)) {
+        qFatal("failed to open %s for reading", parser.positionalArguments().first().toLocal8Bit().data());
+        exit(2);
+    }
+    QFile outFile(parser.positionalArguments().at(1));
+    if (!outFile.open(QIODevice::WriteOnly)) {
+        qFatal("failed to open %s for writing", parser.positionalArguments().at(1).toLocal8Bit().data());
+        exit(2);
+    }
+    QTextDocument doc;
+    doc.setHtml(QString::fromUtf8(inFile.readAll()));
+    outFile.write(doc.toMarkdown().toUtf8());
+}
diff --git a/tests/manual/markdown/html2md.pro b/tests/manual/markdown/html2md.pro
new file mode 100644
index 0000000000..4d6254e5a0
--- /dev/null
+++ b/tests/manual/markdown/html2md.pro
@@ -0,0 +1,6 @@
+TEMPLATE = app
+TARGET = html2md
+INCLUDEPATH += .
+#QT += gui-private
+SOURCES += html2md.cpp
+