opus_pvq: improve PVQ search for low Ks

Since the probelm mentioned only happened when the phase was negative
(e.g. the sum had to be decreased), only discarding dimensions with a
zero pulse in that case restored the search's previously low distortion
at low Ks when the phase is never negative.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

1 files changed, 5 insertions, 2 deletions

diff --git a/libavcodec/opus_pvq.c b/libavcodec/opus_pvq.c
index 5920ab0ed1..0d3536718d 100644
--- a/libavcodec/opus_pvq.c
+++ b/libavcodec/opus_pvq.c
@@ -413,11 +413,14 @@ static void celt_pvq_search(float *X, int *y, int K, int N)
         y_norm += 1.0f;
 
         for (i = 0; i < N; i++) {
+            /* If the sum has been overshot and the best place has 0 pulses allocated
+             * to it, attempting to decrease it further will actually increase the
+             * sum. Prevent this by disregarding any 0 positions when decrementing. */
+            const int ca = 1 ^ ((y[i] == 0) & (phase < 0));
             float xy_new = xy_norm + 1*phase*FFABS(X[i]);
             float y_new  = y_norm  + 2*phase*FFABS(y[i]);
             xy_new = xy_new * xy_new;
-            /* FIXME: the y[i] check makes the search slightly worse at Ks below 5 */
-            if (y[i] && (max_den*xy_new) > (y_new*max_num)) {
+            if (ca && (max_den*xy_new) > (y_new*max_num)) {
                 max_den = y_new;
                 max_num = xy_new;
                 max_idx = i;