avcodec/x86/mathops: clip constants used with shift instructions within inline assembly

Fixes assembling with binutil as >= 2.41

Signed-off-by: James Almer <jamrial@gmail.com>
(cherry picked from commit effadce6c756247ea8bae32dc13bb3e6f464f0eb)

• [DH] libavcodec/x86/mathops.h

avformat/mov: Check if a key is longer than the atom containing it

Stop reading keys and return AVERROR_INVALIDDATA if key_size
is larger than the amount of space left in the atom.

Bug: https://crbug.com/41496983
Signed-off-by: Eugene Zemtsov <eugene@chromium.org>
Signed-off-by: James Almer <jamrial@gmail.com>
(cherry picked from commit 8a23a145d85964950123952d897b89c2c2b1b8c5)

• [DH] libavformat/mov.c

avcodec/nvdec: reset bitstream_len/nb_slices when resetting bitstream pointer

• [DH] libavcodec/nvdec.c

avcodec/noise_bsf: Check for wrapped frames

Wrapped frames contain pointers so they need specific code to
noise them, the generic code would lead to segfaults

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 0889ebc577749ee6abc620bc9030d2002487935f)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/noise_bsf.c

Changelog: update

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] Changelog

avformat/nutdec: Add check for avformat_new_stream

Check for failure of avformat_new_stream() and propagate
the error code.

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavformat/nutdec.c

Update for 4.1.11

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] Changelog
• [DH] RELEASE
• [DH] doc/Doxyfile

avformat/wavdec: Check that smv block fits in available space

Fixes: OOM
Fixes: 56271/clusterfuzz-testcase-minimized-ffmpeg_dem_WAV_fuzzer-5290810045497344

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a76efafdb9be966ae3ad52b32370dc644dd582bf)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavformat/wavdec.c

avcodec/tak: Check remaining bits in ff_tak_decode_frame_header()

Fixes: out of array access
Fixes: 45982/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_TAK_fuzzer-6682195323650048

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 19b66b89da4b4ff086dc1fc79bbf540e82bdbcb4)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/tak.c

avcodec/utils: the IFF_ILBM implementation assumes that there are a multiple of 16 allocated

Fixes: out of array access
Fixes: 45982/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_IFF_ILBM_fuzzer-5124452659888128
Fixes: 45982/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_IFF_ILBM_fuzzer-6362836707442688

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 34056a94eab5f8fbc7e0b8510f7c9851931f23b7)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/utils.c

avcodec/pngdec: Do not pass AVFrame into global header decode

The global header should not contain a frame, and decoding it
would result in leaks

Fixes: memleak
Fixes: 45982/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APNG_fuzzer-6603443149340672

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit d31d4f32283f765c79d6e127d31ee2c37a0acef7)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/pngdec.c

avcodec/vorbisdec: Check codebook float values to be finite

Fixes: Timeout
Fixes: 55116/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VORBIS_fuzzer-4572159970508800

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit cadd7e7a7589b5c118ad1648a09c629a6b65a3be)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/vorbisdec.c

avcodec/g2meet: Replace fake allocation avoidance for framebuf

framebuf is only allocated when the new width/height are larger than the old
but nothing sets the old so its always allocated.
Use av_fast_mallocz() instead.

Fixes: Timeout
Fixes: 55094/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_G2M_fuzzer-5116909932904448

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 38adbc6eebd7f2f34ecf1b0b18019e88bad9d9f4)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/g2meet.c

avcodec/lcldec: More space for rgb24

Fixes: Ticket 10239
Fixes: zlib_306_306_rgb24.av

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit e2c3aa8e2b800c5b860315277b3ea426b8b23393)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/lcldec.c

avcodec/lcldec: Support 4:1:1 and 4:2:2 with odd width

Fixes: Ticket10240
Fixes: zlib_306_306_yuv422.avi
Fixes: zlib_306_306_yuv411.avi

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 0cf1ac905d2d97355a389c3baa4e132824b29f21)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/lcldec.c

libavcodec/lcldec: width and height should not be unsigned

Computations like col < width - 3 will not work with unsigned width=1

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 3eb4e28c26c3bce608214f392ab1fe6ee28ec1df)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/lcldec.c

avcodec/escape124: Check that blocks are allocated before use

Fixes: NULL pointer dereference
Fixes: 57819/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ESCAPE124_fuzzer-5077280228769792

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 5366ae12b9ba60404822f6b39b41f6c0d98a7c8a)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/escape124.c

avcodec/huffyuvdec: Fix undefined behavior with shift

Fixes: left shift of negative value -1
Fixes: 57554/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_FFVHUFF_fuzzer-4853603839115264

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 27e7857bd1127974ffe1512293abee83b1035194)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/huffyuvdec.c

avcodec/j2kenc: Replace RGB24 special case by generic test

This fixes RGB48 with libavcodec as decoder

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit ad4d647591dbd953a5cf3a32a779ee5e42465bbb)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/j2kenc.c

avcodec/j2kenc: remove misleading pred value

This field is only checked for being 0 or not and not zero means 5/3

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 0adb375377f369b69b24d86bbfe674b7693ccf3c)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

• [DH] libavcodec/j2kenc.c