Improve ffmpeg hw accel and codecs finding

A bunch of the improvements mostly related to linux.
The most problem with linux hw acceleration is that there're no exact
working acceleration for all hardwares.

On linux we had only hw acceleration 'vaapi' that requires
installation of vaapi drivers, also on some hardwares
it's not supported or not stable.
Pretty often ffmpeg returns vaapi as supported hw device but
it failes on creation vaapi hw context.
The same might happen with other accelerations as well.

The commit makes selection and creation of hw devices
with codec for them more reliable and flexible,
also, it extends the list of possible hw accel devices for linux,
and makes customization of codecs choosing easier in the future.

The functionality has been checked with cuda and vdpau hw accelerations
on PC with nvidia rtx a2000. Both decoding and encoding work fine,
handling of vdpau textures is to be done in the future.
The hope and prospective is that will work on
embedded platforms with nvidia gpu, it is to be tested afterwards.

Since hw accel types are very platform specific thing,
I propose runtime env vars
QT_FFMPEG_ENCODING_HW_DEVICE_TYPES and QT_FFMPEG_DECODING_HW_DEVICE_TYPES
to allow advanced users configuring priorities of used acceleration for
the exact target platform.

The changes have been tested on linux, macos, windows.

Should be done afterwards:
- conversion vdpau textures to opengl, QTBUG-111593
- add vaapi, vdpau, cuda to linux CI builds. I've checked that their
  adding don't affect ffmpeg lib size to much, less than 1%.
- describe hw acceleration details and some setup guide
  in the qt documentation.
- handle codec open fail during codecs search.

Pick-to: 6.5 6.5.0
Change-Id: I8f120f53573f928a0673ed4ad94607a0ac57822d
Reviewed-by: Lars Knoll <lars@knoll.priv.no>

1 files changed, 40 insertions, 48 deletions

diff --git a/src/plugins/multimedia/ffmpeg/qffmpegvideoframeencoder.cpp b/src/plugins/multimedia/ffmpeg/qffmpegvideoframeencoder.cpp
index 6985ffea4..b72f9ceac 100644
--- a/src/plugins/multimedia/ffmpeg/qffmpegvideoframeencoder.cpp
+++ b/src/plugins/multimedia/ffmpeg/qffmpegvideoframeencoder.cpp
@@ -41,59 +41,44 @@ VideoFrameEncoder::VideoFrameEncoder(const QMediaEncoderSettings &encoderSetting
     d->sourceFormat = sourceFormat;
     d->sourceSWFormat = swFormat;
 
-    auto qVideoCodec = encoderSettings.videoCodec();
-    auto codecID = QFFmpegMediaFormatInfo::codecIdForVideoCodec(qVideoCodec);
-
-#ifndef QT_DISABLE_HW_ENCODING
-    auto [preferredTypes, size] = HWAccel::preferredDeviceTypes();
-    for (qsizetype i = 0; i < size; i++) {
-        auto accel = HWAccel::create(preferredTypes[i]);
-        if (!accel)
-            continue;
-
-        auto matchesSizeConstraints = [&]() -> bool {
-            auto *constraints = av_hwdevice_get_hwframe_constraints(accel->hwDeviceContextAsBuffer(), nullptr);
-            if (!constraints)
-                return true;
-                // Check size constraints
-            bool result = (d->sourceSize.width() >= constraints->min_width && d->sourceSize.height() >= constraints->min_height &&
-                           d->sourceSize.width() <= constraints->max_width && d->sourceSize.height() <= constraints->max_height);
-            av_hwframe_constraints_free(&constraints);
-            return result;
-        };
+    const auto qVideoCodec = encoderSettings.videoCodec();
+    const auto codecID = QFFmpegMediaFormatInfo::codecIdForVideoCodec(qVideoCodec);
+
+    auto matchesSizeConstraints = [&sourceSize](const HWAccel &accel) {
+        AVHWFramesConstraintsUPtr constraints(
+                av_hwdevice_get_hwframe_constraints(accel.hwDeviceContextAsBuffer(), nullptr));
+        if (!constraints)
+            return true;
+
+        return sourceSize.width() >= constraints->min_width
+                && sourceSize.height() >= constraints->min_height
+                && sourceSize.width() <= constraints->max_width
+                && sourceSize.height() <= constraints->max_height;
+    };
 
-        if (!matchesSizeConstraints())
-            continue;
+    std::tie(d->codec, d->accel) = HWAccel::findEncoderWithHwAccel(codecID, matchesSizeConstraints);
 
-        d->codec = accel->hardwareEncoderForCodecId(codecID);
-        if (!d->codec)
-            continue;
-        d->accel = std::move(accel);
-        break;
-    }
-#endif
-
-    if (!d->accel) {
-        d->codec = avcodec_find_encoder(codecID);
-        if (!d->codec) {
-            qWarning() << "Could not find encoder for codecId" << codecID;
-            d = {};
-            return;
-        }
+    if (!d->codec)
+        d->codec = findAVEncoder(codecID, {}, d->targetFormat);
+
+    if (!d->codec)
+        d->codec = findAVEncoder(codecID, {}, swFormat);
+
+    if (!d->codec)
+        d->codec = findAVEncoder(codecID);
+
+    if (!d->codec) {
+        qWarning() << "Could not find encoder for codecId" << codecID;
+        d = {};
+        return;
     }
-    auto supportsFormat = [&](AVPixelFormat fmt) {
-        if (auto fmts = d->codec->pix_fmts) {
-            for (; *fmts != AV_PIX_FMT_NONE; ++fmts)
-                if (*fmts == fmt)
-                    return true;
-        }
-        return false;
-    };
+
+    qCDebug(qLcVideoFrameEncoder) << "found encoder" << d->codec->name << "for id" << d->codec->id;
 
     d->targetFormat = d->sourceFormat;
 
-    if (!supportsFormat(d->sourceFormat)) {
-        if (supportsFormat(swFormat))
+    if (!isAVFormatSupported(d->codec, d->sourceFormat)) {
+        if (isAVFormatSupported(d->codec, swFormat))
             d->targetFormat = swFormat;
         else if (d->codec->pix_fmts)
             // Take first format the encoder supports. Might want to improve upon this
@@ -382,7 +367,14 @@ AVPacket *VideoFrameEncoder::retrievePacket()
         return nullptr;
     }
     auto ts = timeStampMs(packet->pts, d->stream->time_base);
-    qCDebug(qLcVideoFrameEncoder) << "got a packet" << packet->pts << (ts ? *ts : 0);
+
+    qCDebug(qLcVideoFrameEncoder) << "got a packet" << packet->pts << packet->dts << (ts ? *ts : 0);
+
+    if (packet->dts != AV_NOPTS_VALUE && packet->pts < packet->dts) {
+        // the case seems to be an ffmpeg bug
+        packet->dts = AV_NOPTS_VALUE;
+    }
+
     packet->stream_index = d->stream->id;
     return packet;
 }