vdpau: do not use buggy HEVC support by default

NVIDIA broke its own API when using VDPAU decoding. If you retrieve the
decoded YUV data, or if you map the surfaces with GL interop, the result
are interlacing artifacts. The only way to get non-broken data is by
using the vdpau video mixer to convert it to RGB. There is no way to
block the non-working operations in a reasonable way (a VdpVideoSurface
has to support all operations).

NVIDIA refuses to fix this issue (they "fixed" it by making it work with
the video mixer, but the rest is still broken). There is no sign of that
changing.

Do not use HEVC by default with the generic hwaccle API. Detect whether
it's the NVIDIA native implementation, and exit with an error. (The same
thing work with the MESA implementation.)

As an escape hatch and to allow applications to use the decoder if they
really want to (perhaps because they make sure to explicitly use the
video mixer), reuse AV_HWACCEL_FLAG_ALLOW_PROFILE_MISMATCH to disable
this check.

Once NVIDIA fixes the bug, working driver versions could be detected,
and it could be allowed again.

1 files changed, 19 insertions, 0 deletions

diff --git a/libavcodec/vdpau.c b/libavcodec/vdpau.c
index 9c7804a287..42ebddbeee 100644
--- a/libavcodec/vdpau.c
+++ b/libavcodec/vdpau.c
@@ -127,6 +127,8 @@ int ff_vdpau_common_init(AVCodecContext *avctx, VdpDecoderProfile profile,
     VdpVideoSurfaceQueryCapabilities *surface_query_caps;
     VdpDecoderQueryCapabilities *decoder_query_caps;
     VdpDecoderCreate *create;
+    VdpGetInformationString *info;
+    const char *info_string;
     void *func;
     VdpStatus status;
     VdpBool supported;
@@ -209,6 +211,23 @@ int ff_vdpau_common_init(AVCodecContext *avctx, VdpDecoderProfile profile,
         return AVERROR(ENOTSUP);
 
     status = vdctx->get_proc_address(vdctx->device,
+                                     VDP_FUNC_ID_GET_INFORMATION_STRING,
+                                     &func);
+    if (status != VDP_STATUS_OK)
+        return vdpau_error(status);
+    else
+        info = func;
+
+    status = info(&info_string);
+    if (status != VDP_STATUS_OK)
+        return vdpau_error(status);
+    if (avctx->codec_id == AV_CODEC_ID_HEVC && strncmp(info_string, "NVIDIA ", 7) == 0 &&
+        !(avctx->hwaccel_flags & AV_HWACCEL_FLAG_ALLOW_PROFILE_MISMATCH)) {
+        av_log(avctx, AV_LOG_VERBOSE, "HEVC with NVIDIA VDPAU drivers is buggy, skipping.\n");
+        return AVERROR(ENOTSUP);
+    }
+
+    status = vdctx->get_proc_address(vdctx->device,
                                      VDP_FUNC_ID_VIDEO_SURFACE_QUERY_CAPABILITIES,
                                      &func);
     if (status != VDP_STATUS_OK)