Harden video processing pipeline for out-of-range signals

This patch makes the video processing pipeline well-behaved when facing
out-of-range video signals. It replaces gamma 2.2 approximation of sRGB
with a proper sRGB transfer function as described in
https://en.wikipedia.org/wiki/SRGB, and saturates output RGB signal to
be within the range of the underlying texture type. Together, these two
changes prevents Infinity or NaN values in video output, and prevents
potential wrap-around with out-of-range values.

Conversion from linear space to sRGB using an inverse gamma 2.2 transfer
function is problematic because x^(1/2.2) is imaginary for negative x.
In particular, the openGL pow(x, y) is undefined for negative x. The new
equations are linear for low input values, and therefore well behaved
with negative input. Now that we have well-defined RGB output also with
out-of-range values, we can safely clamp any out-of-range value to be
within the expected [0.0, 1.0] range.

Out-of-range inputs can be caused by numerical inaccuracies when
converting from YUV to RGB, and from out-of-range values in the input
YUV video signal. In particular with 'video' range media, values outside
the 16-235 range is expected and must be handled gracefully.

Fixes: QTBUG-113317
Pick-to: 6.5 6.6 6.7
Change-Id: Ia62777650efc19abf7e6f2a2a65f20deb0ec2a1c
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Lars Knoll <lars@knoll.priv.no>
Reviewed-by: Pavel Dubsky <pavel.dubsky@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Artem Dyomin <artem.dyomin@qt.io>

1 files changed, 14 insertions, 4 deletions

diff --git a/src/multimedia/shaders/colortransfer.glsl b/src/multimedia/shaders/colortransfer.glsl
index 76af51b6b..1d70ae9eb 100644
--- a/src/multimedia/shaders/colortransfer.glsl
+++ b/src/multimedia/shaders/colortransfer.glsl
@@ -12,14 +12,24 @@ vec4 convertRec709ToLinear(vec4 rgba)
     return mix(high, low, cutoff);
 }
 
-vec4 convertSRGBToLinear(vec4 rgba)
+vec4 convertSRGBToLinear(vec4 sRGB)
 {
-    return pow(rgba, vec4(2.2));
+    // https://en.wikipedia.org/wiki/SRGB
+    const bvec3 cutoff = lessThanEqual(sRGB.rgb, vec3(0.04045));
+    const vec3 low = sRGB.rgb / 12.92;
+    const vec3 high = pow((sRGB.rgb + 0.055) / 1.055, vec3(2.4));
+    vec3 linear = mix(high, low, cutoff);
+    return vec4(linear, sRGB.a);
 }
 
-vec4 convertSRGBFromLinear(vec4 rgba)
+vec4 convertSRGBFromLinear(vec4 linear)
 {
-    return pow(rgba, vec4(1./2.2));
+    // https://en.wikipedia.org/wiki/SRGB
+    const bvec3 cutoff = lessThanEqual(linear.rgb, vec3(0.0031308));
+    const vec3 low = linear.rgb * 12.92;
+    const vec3 high = 1.055 * pow(linear.rgb, vec3(1.0 / 2.4f)) - 0.055;
+    vec3 sRGB = mix(high, low, cutoff);
+    return vec4(sRGB, linear.a);
 }
 
 // This uses the PQ transfer function, see also https://en.wikipedia.org/wiki/Perceptual_quantizer