avcodec/atrac9dec: Don't create VLCs that are never used

The ATRAC9 decoder creates VLCs with parameters contained in
HuffmanCodebooks; some of these HuffmanCodebooks are empty and yet
VLCs (that were completely unused*) were created from them. Said VLC
contained a single table with 512 VLC_TYPE[2] entries, each of which
indicated that this is an invalid code. This commit stops creating said
VLCs.

*: read_coeffs_coarse() uses the HuffmanCodebook
at9_huffman_coeffs[cb][prec][cbi]. prec is c->precision_coarse[i] + 1
and every precision_coarse entry is in the 1..15 range after
calc_precision(), so prec is >= 2 (all codebooks with prec < 2 are
empty). The remaining empty codebooks are those with cb == 1 and cbi ==
0, yet this is impossible, too: cb is given by c->codebookset[i] and
this is always 0 if i < 8 (because those are never set to anything else
in calc_codebook_idx()) and cbi is given by at9_q_unit_to_codebookidx[i]
which is never zero if i >= 8.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

1 files changed, 2 insertions, 2 deletions

diff --git a/libavcodec/atrac9dec.c b/libavcodec/atrac9dec.c
index b7a46784fb..c7c6a12f01 100644
--- a/libavcodec/atrac9dec.c
+++ b/libavcodec/atrac9dec.c
@@ -964,8 +964,8 @@ static av_cold int atrac9_decode_init(AVCodecContext *avctx)
     /* Coefficient VLCs */
     tab = at9_coeffs_tab;
     for (int i = 0; i < 2; i++) {
-        for (int j = 0; j < 8; j++) {
-            for (int k = 0; k < 4; k++) {
+        for (int j = 2; j < 8; j++) {
+            for (int k = i; k < 4; k++) {
                 const HuffmanCodebook *hf = &at9_huffman_coeffs[i][j][k];
                 ret = ff_init_vlc_from_lengths(&s->coeff_vlc[i][j][k], 9,
                                                hf->size, &tab[0][1], 2,