avcodec/nvenc: Declare support for P016

nvenc doesn't support P016, but we have two problems today:

1) We declare support for YUV444P16 which nvenc also doesn't support.
   We do this because it's the only pix_fmt we have that can
   approximate nvenc's internal format that is YUV444P10 with data in
   MSBs instead of LSBs. Because the declared format is a 16bit one,
   it will be preferrentially chosen when encoding >10bit content,
   but that content will normally be YUV420P12 or P016 which should
   get mapped to P010 and not YUV444P10.

2) Transcoding P016 content with nvenc should be possible in a pure
   hardware pipeline, and that can't be done if nvenc doesn't say it
   accepts P016. By mapping it to P010, we can use it, albeit with
   truncation. I have established that swscale doesn't know how to
   dither to 10bits so we'd get truncation anyway, even if we tried
   to do this 'properly'.

1 files changed, 4 insertions, 1 deletions

diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
index 39c3aa1fbb..1f601a63bd 100644
--- a/libavcodec/nvenc.c
+++ b/libavcodec/nvenc.c
@@ -41,7 +41,8 @@ const enum AVPixelFormat ff_nvenc_pix_fmts[] = {
     AV_PIX_FMT_NV12,
     AV_PIX_FMT_P010,
     AV_PIX_FMT_YUV444P,
-    AV_PIX_FMT_YUV444P16,
+    AV_PIX_FMT_P016,      // Truncated to 10bits
+    AV_PIX_FMT_YUV444P16, // Truncated to 10bits
     AV_PIX_FMT_0RGB32,
     AV_PIX_FMT_0BGR32,
     AV_PIX_FMT_CUDA,
@@ -52,6 +53,7 @@ const enum AVPixelFormat ff_nvenc_pix_fmts[] = {
 };
 
 #define IS_10BIT(pix_fmt)  (pix_fmt == AV_PIX_FMT_P010    || \
+                            pix_fmt == AV_PIX_FMT_P016    || \
                             pix_fmt == AV_PIX_FMT_YUV444P16)
 
 #define IS_YUV444(pix_fmt) (pix_fmt == AV_PIX_FMT_YUV444P || \
@@ -1219,6 +1221,7 @@ static NV_ENC_BUFFER_FORMAT nvenc_map_buffer_format(enum AVPixelFormat pix_fmt)
     case AV_PIX_FMT_NV12:
         return NV_ENC_BUFFER_FORMAT_NV12_PL;
     case AV_PIX_FMT_P010:
+    case AV_PIX_FMT_P016:
         return NV_ENC_BUFFER_FORMAT_YUV420_10BIT;
     case AV_PIX_FMT_YUV444P:
         return NV_ENC_BUFFER_FORMAT_YUV444_PL;