a11y: Add support for Tables in macOS bridge

For a11y purposes, a table needs to be mapped into a logical
accessibility hierarchy.  There are several ways of doing this mapping,
and unfortunately macOS expects something different than what
QAccessibleInterface does.

So suppose we have a a 2x2 QTableView with both horizontal and vertical
header like this (the names reflect the QAccessible::Role names):

  +-----------+--------------+--------------+
  |           | ColumnHeader | ColumnHeader |
  +-----------+--------------+--------------+
  | RowHeader | Cell         | Cell         |
  +-----------+--------------+--------------+
  | RowHeader | Cell         | Cell         |
  +-----------+--------------+--------------+

In order to be presented to the screen reader on a platform, it goes
through two rounds of mapping:

QAccessibleInterface will have all headers and cells as direct children of the table:

 - Table
   +- ColumnHeader
   +- ColumnHeader
   +- RowHeader
   +- Cell
   +- Cell
   +- RowHeader
   +- Cell
   +- Cell

macOS expects a deeper hierarchy:

 - AXTable [QAccessible::Table]
   +- AXRow     [Qt:no eqiuivalent]
      +- [QAccessible::Cell] (The content of the cell, e.g. AXButton, AXGroup or whatever)
      +- [QAccessible::Cell] (The content of the cell, e.g. AXButton, AXGroup or whatever)
   +- AXRow
      +- [QAccessible::Cell] (The content of the cell, e.g. AXButton, AXGroup or whatever)
      +- [QAccessible::Cell] (The content of the cell, e.g. AXButton, AXGroup or whatever)
   +- AXColumn (this seems to just store the geometry of the column)
   +- AXColumn (this seems to just store the geometry of the column)
   +- AXGroup   (this represents the column headers)
      +- AXSortButton (clicking a header cell will trigger sorting)
      +- AXSortButton (clicking a header cell will trigger sorting)

It's unclear to me how RowHeaders are mapped (they are rarer than
ColumnHeaders, I expect to find them in e.g. spreadsheet applications).
I haven't found any native usage of them. So this patch simply ignores
them.

Notice that macOS have a three layer deep hierarchy to represent a table
(Table->Row->Cell), while QAccessibleInterface has a two-layer deep
hierarchy (Table->Row/Cell).

In the macOS bridge we therefore need to "inject" the Row/Column element
to be "between" the table and the cell.

The table will take ownership of all row and column elements that are
children of the table. These elements are not inserted into the cache
(it would be pointless, since the cache is basically just a mapping
between the QAccessibleInterface -> QMacAccessibilityElement, and the
row and column elements does not have a corresponding
QAccessibleInterface to be mapped from).

The rows and columns are therefore also created as soon as the table
element is initialized, and they are stored in two NSMutableArray
members of QMacAccessibilityElement.

A table is constructed like any other accessibility element, with a
valid axid and synthesizedRole set to nil.
Each child row and column element is constructed with the same axid as the
parent table element, and they will have the synthesizedRole set to
either NSAccessibilityRow or NSAccessibilityColumn.
With the synthesizedRole member we can then identify if we are a row,
column or the actual table, and implement their respective behaviors.

Notice that the child row/column is created with the parent's table axid
in order for them to have a way of finding their parent table element.
(there is no 'parent' member variable in QMacAccessibilityElement)

This glorious scheme isn't pretty, but seems to work.

Fixes: QTBUG-37207
Change-Id: I7c2451e629f5331b9a0ed61dc22c6e74a82cc173
Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>

1 files changed, 255 insertions, 2 deletions

diff --git a/src/plugins/platforms/cocoa/qcocoaaccessibilityelement.mm b/src/plugins/platforms/cocoa/qcocoaaccessibilityelement.mm
index f3c3d0bfaa..68e7947162 100644
--- a/src/plugins/platforms/cocoa/qcocoaaccessibilityelement.mm
+++ b/src/plugins/platforms/cocoa/qcocoaaccessibilityelement.mm
@@ -80,14 +80,37 @@ static void convertLineOffset(QAccessibleTextInterface *text, int *line, int *of
 
 @implementation QMacAccessibilityElement {
     QAccessible::Id axid;
+
+    // used by NSAccessibilityTable
+    NSMutableArray<QMacAccessibilityElement *> *rows;       // corresponds to accessibilityRows
+    NSMutableArray<QMacAccessibilityElement *> *columns;    // corresponds to accessibilityColumns
+
+    // If synthesizedRole is set, this means that this objects does not have a corresponding
+    // QAccessibleInterface, but it is synthesized by the cocoa plugin in order to meet the
+    // NSAccessibility requirements.
+    // The ownership is controlled by the parent object identified with the axid member variable.
+    // (Therefore, if this member is set, this objects axid member is the same as the parents axid
+    // member)
+    NSString *synthesizedRole;
 }
 
 - (instancetype)initWithId:(QAccessible::Id)anId
 {
+    return [self initWithId:anId role:nil];
+}
+
+- (instancetype)initWithId:(QAccessible::Id)anId role:(NSAccessibilityRole)role
+{
     Q_ASSERT((int)anId < 0);
     self = [super init];
     if (self) {
         axid = anId;
+        rows = nil;
+        columns = nil;
+        synthesizedRole = role;
+        QAccessibleInterface *iface = QAccessible::accessibleInterface(axid);
+        if (iface && iface->tableInterface() && !synthesizedRole)
+            [self updateTableModel];
     }
 
     return self;
@@ -115,18 +138,26 @@ static void convertLineOffset(QAccessibleTextInterface *text, int *line, int *of
 
 - (void)invalidate {
     axid = 0;
+    rows = nil;
+    columns = nil;
+    synthesizedRole = nil;
+
     NSAccessibilityPostNotification(self, NSAccessibilityUIElementDestroyedNotification);
     [self release];
 }
 
 - (void)dealloc {
+    if (rows)
+        [rows release]; // will also release all entries first
+    if (columns)
+        [columns release]; // will also release all entries first
     [super dealloc];
 }
 
 - (BOOL)isEqual:(id)object {
     if ([object isKindOfClass:[QMacAccessibilityElement class]]) {
         QMacAccessibilityElement *other = object;
-        return other->axid == axid;
+        return other->axid == axid && other->synthesizedRole == synthesizedRole;
     } else {
         return NO;
     }
@@ -136,6 +167,50 @@ static void convertLineOffset(QAccessibleTextInterface *text, int *line, int *of
     return axid;
 }
 
+- (BOOL)isManagedByParent {
+    return synthesizedRole != nil;
+}
+
+- (NSMutableArray *)populateTableArray:(NSMutableArray *)array role:(NSAccessibilityRole)role count:(int)count
+{
+    QAccessibleInterface *iface = QAccessible::accessibleInterface(axid);
+    if (iface && iface->isValid()) {
+        if (!array) {
+            array = [NSMutableArray<QMacAccessibilityElement *> arrayWithCapacity:count];
+            [array retain];
+        } else {
+            [array removeAllObjects];
+        }
+        Q_ASSERT(array);
+        for (int n = 0; n < count; ++n) {
+            // columns will have same axid as table (but not inserted in cache)
+            QMacAccessibilityElement *element =
+                    [[QMacAccessibilityElement alloc] initWithId:axid role:role];
+            if (element) {
+                [array addObject:element];
+                [element release];
+            } else {
+                qWarning("QCocoaAccessibility: invalid child");
+            }
+        }
+        return array;
+    }
+    return nil;
+}
+
+
+- (void)updateTableModel
+{
+    QAccessibleInterface *iface = QAccessible::accessibleInterface(axid);
+    if (iface && iface->isValid()) {
+        if (QAccessibleTableInterface *table = iface->tableInterface()) {
+            Q_ASSERT(!self.isManagedByParent);
+            rows = [self populateTableArray:rows role:NSAccessibilityRowRole count:table->rowCount()];
+            columns = [self populateTableArray:columns role:NSAccessibilityColumnRole count:table->columnCount()];
+        }
+    }
+}
+
 //
 // accessibility protocol
 //
@@ -168,6 +243,8 @@ static void convertLineOffset(QAccessibleTextInterface *text, int *line, int *of
     QAccessibleInterface *iface = QAccessible::accessibleInterface(axid);
     if (!iface || !iface->isValid())
         return NSAccessibilityUnknownRole;
+    if (synthesizedRole)
+        return synthesizedRole;
     return QCocoaAccessible::macRole(iface);
 }
 
@@ -189,6 +266,81 @@ static void convertLineOffset(QAccessibleTextInterface *text, int *line, int *of
     QAccessibleInterface *iface = QAccessible::accessibleInterface(axid);
     if (!iface || !iface->isValid())
         return nil;
+    if (QAccessibleTableInterface *table = iface->tableInterface()) {
+        // either a table or table rows/columns
+        if (!synthesizedRole) {
+            // This is the table element, parent of all rows and columns
+            /*
+             * Typical 2x2 table hierarchy as can be observed in a table found under
+             *   Apple -> System Settings -> General -> Login Items    (macOS 13)
+             *
+             *  (AXTable)
+             *  | Columns: NSArray* (2 items)
+             *  | Rows: NSArray* (2 items)
+             *  | Visible Columns: NSArray* (2 items)
+             *  | Visible Rows: NSArray* (2 items)
+             *  | Children: NSArray* (5 items)
+        +----<--| Header: (AXGroup)
+        |    *  +-- (AXRow)
+        |    *  |   +-- (AXText)
+        |    *  |   +-- (AXTextField)
+        |    *  +-- (AXRow)
+        |    *  |   +-- (AXText)
+        |    *  |   +-- (AXTextField)
+        |    *  +-- (AXColumn)
+        |    *  |     Header: "Item" (sort button)
+        |    *  |     Index: 0
+        |    *  |     Rows: NSArray* (2 items)
+        |    *  |     Visible Rows: NSArray* (2 items)
+        |    *  +-- (AXColumn)
+        |    *  |     Header: "Kind" (sort button)
+        |    *  |     Index: 1
+        |    *  |     Rows: NSArray* (2 items)
+        |    *  |     Visible Rows: NSArray* (2 items)
+        +---->  +-- (AXGroup)
+             *      +-- (AXButton/AXSortButton) Item [NSAccessibilityTableHeaderCellProxy]
+             *      +-- (AXButton/AXSortButton) Kind [NSAccessibilityTableHeaderCellProxy]
+             */
+            NSArray *rs = [self accessibilityRows];
+            NSArray *cs = [self accessibilityColumns];
+            const int rCount = int([rs count]);
+            const int cCount = int([cs count]);
+            int childCount = rCount + cCount;
+            NSMutableArray<QMacAccessibilityElement *> *tableChildren =
+                    [NSMutableArray<QMacAccessibilityElement *> arrayWithCapacity:childCount];
+            for (int i = 0; i < rCount; ++i) {
+                [tableChildren addObject:[rs objectAtIndex:i]];
+            }
+            for (int i = 0; i < cCount; ++i) {
+                [tableChildren addObject:[cs objectAtIndex:i]];
+            }
+            return NSAccessibilityUnignoredChildren(tableChildren);
+        } else if (synthesizedRole == NSAccessibilityColumnRole) {
+            return nil;
+        } else if (synthesizedRole == NSAccessibilityRowRole) {
+            // axid matches the parent table axid so that we can easily find the parent table
+            // children of row are cell/any items
+            QMacAccessibilityElement *tableElement = [QMacAccessibilityElement elementWithId:axid];
+            Q_ASSERT(tableElement->rows);
+            NSUInteger rowIndex = [tableElement->rows indexOfObjectIdenticalTo:self];
+            Q_ASSERT(rowIndex != NSNotFound);
+            int numColumns = table->columnCount();
+            NSMutableArray<QMacAccessibilityElement *> *cells =
+                    [NSMutableArray<QMacAccessibilityElement *> arrayWithCapacity:numColumns];
+            for (int i = 0; i < numColumns; ++i) {
+                QAccessibleInterface *cell = table->cellAt(rowIndex, i);
+                if (cell && cell->isValid()) {
+                    QAccessible::Id cellId = QAccessible::uniqueId(cell);
+                    QMacAccessibilityElement *element =
+                            [QMacAccessibilityElement elementWithId:cellId];
+                    if (element) {
+                        [cells addObject:element];
+                    }
+                }
+            }
+            return NSAccessibilityUnignoredChildren(cells);
+        }
+    }
     return QCocoaAccessible::unignoredChildren(iface);
 }
 
@@ -208,6 +360,8 @@ static void convertLineOffset(QAccessibleTextInterface *text, int *line, int *of
         return nil;
     if (iface->role() == QAccessible::StaticText)
         return nil;
+    if (self.isManagedByParent)
+        return nil;
     return iface->text(QAccessible::Name).toNSString();
 }
 
@@ -223,12 +377,31 @@ static void convertLineOffset(QAccessibleTextInterface *text, int *line, int *of
     if (!iface || !iface->isValid())
         return nil;
 
+    if (self.isManagedByParent) {
+        // axid is the same for the parent element
+        return NSAccessibilityUnignoredAncestor([QMacAccessibilityElement elementWithId:axid]);
+    }
+
     // macOS expects that the hierarchy is:
     // App -> Window -> Children
     // We don't actually have the window reflected properly in QAccessibility.
     // Check if the parent is the application and then instead return the native window.
 
     if (QAccessibleInterface *parent = iface->parent()) {
+        if (parent->tableInterface()) {
+            if (QAccessibleTableCellInterface *cell = iface->tableCellInterface()) {
+                // parent of cell should be row
+                QAccessible::Id parentId = QAccessible::uniqueId(parent);
+                QMacAccessibilityElement *tableElement =
+                        [QMacAccessibilityElement elementWithId:parentId];
+
+                const int rowIndex = cell->rowIndex();
+                if (tableElement->rows && int([tableElement->rows count]) > rowIndex) {
+                    QMacAccessibilityElement *rowElement = tableElement->rows[rowIndex];
+                    return NSAccessibilityUnignoredAncestor(rowElement);
+                }
+            }
+        }
         if (parent->role() != QAccessible::Application) {
             QAccessible::Id parentId = QAccessible::uniqueId(parent);
             return NSAccessibilityUnignoredAncestor([QMacAccessibilityElement elementWithId: parentId]);
@@ -249,7 +422,44 @@ static void convertLineOffset(QAccessibleTextInterface *text, int *line, int *of
     QAccessibleInterface *iface = QAccessible::accessibleInterface(axid);
     if (!iface || !iface->isValid())
         return NSZeroRect;
-    return QCocoaScreen::mapToNative(iface->rect());
+
+    QRect rect;
+    if (self.isManagedByParent) {
+        if (QAccessibleTableInterface *table = iface->tableInterface()) {
+            // Construct the geometry of the Row/Column by looking at the individual table cells
+            // ### Assumes that cells logical coordinates have spatial ordering (e.g finds the
+            // rows width by taking the union between the leftmost item and the rightmost item in
+            // a row).
+            // Otherwise, we have to iterate over *all* cells in a row/columns to
+            // find out the Row/Column geometry
+            const bool isRow = synthesizedRole == NSAccessibilityRowRole;
+            QPoint cellPos;
+            int &row = isRow ? cellPos.ry() : cellPos.rx();
+            int &col = isRow ? cellPos.rx() : cellPos.ry();
+
+            QMacAccessibilityElement *tableElement =
+                    [QMacAccessibilityElement elementWithId:axid];
+            NSArray *tracks = isRow ? tableElement->rows : tableElement->columns;
+            NSUInteger trackIndex = [tracks indexOfObjectIdenticalTo:self];
+            if (trackIndex != NSNotFound) {
+                row = int(trackIndex);
+                if (QAccessibleInterface *firstCell = table->cellAt(cellPos.y(), cellPos.x())) {
+                    rect = firstCell->rect();
+                    col = isRow ? table->columnCount() : table->rowCount();
+                    if (col > 1) {
+                        --col;
+                        if (QAccessibleInterface *lastCell =
+                                    table->cellAt(cellPos.y(), cellPos.x()))
+                            rect = rect.united(lastCell->rect());
+                    }
+                }
+            }
+        }
+    } else {
+        rect = iface->rect();
+    }
+
+    return QCocoaScreen::mapToNative(rect);
 }
 
 - (NSString*)accessibilityLabel {
@@ -614,6 +824,49 @@ static void convertLineOffset(QAccessibleTextInterface *text, int *line, int *of
     return nil;
 }
 
+/*
+ * Support for table
+ */
+- (NSInteger) accessibilityIndex {
+    NSInteger index = 0;
+    QAccessibleInterface *iface = QAccessible::accessibleInterface(axid);
+    if (iface && iface->isValid()) {
+        if (self.isManagedByParent) {
+            // axid matches the parent table axid so that we can easily find the parent table
+            // children of row are cell/any items
+            if (QAccessibleTableInterface *table = iface->tableInterface()) {
+                QMacAccessibilityElement *tableElement = [QMacAccessibilityElement elementWithId:axid];
+                NSArray *track = synthesizedRole == NSAccessibilityRowRole
+                               ? tableElement->rows : tableElement->columns;
+                if (track) {
+                    NSUInteger trackIndex = [track indexOfObjectIdenticalTo:self];
+                    index = (NSInteger)trackIndex;
+                }
+            }
+        }
+    }
+    return index;
+}
+
+- (NSArray *) accessibilityRows {
+    QAccessibleInterface *iface = QAccessible::accessibleInterface(axid);
+    if (iface && iface->isValid() && iface->tableInterface() && !synthesizedRole) {
+        if (rows)
+            return NSAccessibilityUnignoredChildren(rows);
+    }
+    return nil;
+}
+
+- (NSArray *) accessibilityColumns {
+    QAccessibleInterface *iface = QAccessible::accessibleInterface(axid);
+    if (iface && iface->isValid() && iface->tableInterface() && !synthesizedRole) {
+        if (columns)
+            return NSAccessibilityUnignoredChildren(columns);
+    }
+    return nil;
+}
+
 @end
 
 #endif // QT_CONFIG(accessibility)
+