rhi: Unify handling of special cases for scissor and viewport rects

With OpenGL a scissor (or viewport) rectangle is not allowed to have a negative
width or height. Everything else is allowed. This is also the semantic we wish to
keep for QRhiViewport and QRhiScissor.

This raises some problems. For instance, when we do bottom-left - top-left
rectangle conversion, the case of partially out of bounds rects needs to be
taken into account.

Otherwise, Qt Quick ends up in wrong scissoring in certain cases, typically when
the QQuickWindow size is decreased so the content does not fit because that will
then start generating negative x, y scissors for clipping (which is perfectly
valid but the QRhi backends need to be able to deal with it)

Then there is the problem of having to clamp width and height carefully, because
some validation layers for some APIs will reject a viewport or scissor with
partially out of bounds rectangles.

To verify all this, add a new manual test, based on the cubemap one. (cubemap was
chosen because that is an ideal test scene as it fills the viewport completely, and
so it is visually straightforward when a scissor rectangle is moving around over it)

Fixes: QTBUG-78702
Change-Id: I60614836432ea9934fc0dbd0ac7e88931f476542
Reviewed-by: Christian Strømme <christian.stromme@qt.io>

1 files changed, 21 insertions, 11 deletions

diff --git a/src/gui/rhi/qrhi_p_p.h b/src/gui/rhi/qrhi_p_p.h
index 63f27b6de4..be2808549c 100644
--- a/src/gui/rhi/qrhi_p_p.h
+++ b/src/gui/rhi/qrhi_p_p.h
@@ -233,27 +233,37 @@ bool qrhi_toTopLeftRenderTargetRect(const QSize &outputSize, const std::array<T,
                                     T *x, T *y, T *w, T *h)
 {
     // x,y are bottom-left in QRhiScissor and QRhiViewport but top-left in
-    // Vulkan/Metal/D3D. We also need proper clamping since some
-    // validation/debug layers are allergic to out of bounds scissor or
-    // viewport rects.
+    // Vulkan/Metal/D3D. Our input is an OpenGL-style scissor rect where both
+    // negative x or y, and partly or completely out of bounds rects are
+    // allowed. The only thing the input here cannot have is a negative width
+    // or height. We must handle all other input gracefully, clamping to a zero
+    // width or height rect in the worst case, and ensuring the resulting rect
+    // is inside the rendertarget's bounds because some APIs' validation/debug
+    // layers are allergic to out of bounds scissor or viewport rects.
 
     const T outputWidth = outputSize.width();
     const T outputHeight = outputSize.height();
     const T inputWidth = r[2];
     const T inputHeight = r[3];
 
-    *x = qMax<T>(0, r[0]);
-    *y = qMax<T>(0, outputHeight - (r[1] + inputHeight));
-    *w = inputWidth;
-    *h = inputHeight;
-
-    if (*x >= outputWidth || *y >= outputHeight)
+    if (inputWidth < 0 || inputHeight < 0)
         return false;
 
+    *x = r[0];
+    *y = outputHeight - (r[1] + inputHeight);
+
+    const T widthOffset = *x < 0 ? -*x : 0;
+    const T heightOffset = *y < 0 ? -*y : 0;
+
+    *x = qBound<T>(0, *x, outputWidth - 1);
+    *y = qBound<T>(0, *y, outputHeight - 1);
+    *w = qMax<T>(0, inputWidth - widthOffset);
+    *h = qMax<T>(0, inputHeight - heightOffset);
+
     if (*x + *w > outputWidth)
-        *w = outputWidth - *x;
+        *w = qMax<T>(0, outputWidth - *x - 1);
     if (*y + *h > outputHeight)
-        *h = outputHeight - *y;
+        *h = qMax<T>(0, outputHeight - *y - 1);
 
     return true;
 }