From b7fb7c4542af63fea433a5417e4efe2d8c4422f6 Mon Sep 17 00:00:00 2001 From: Ganesh Ajjanagadde Date: Wed, 28 Oct 2015 22:02:25 -0400 Subject: avutil/mathematics: make av_gcd more robust This ensures that no undefined behavior is invoked, while retaining identical return values in all cases and at no loss of performance (identical asm on clang and gcc). Essentially, this patch exchanges undefined behavior with implementation defined behavior, a strict improvement. Rationale: 1. The ideal solution is to have the return type a uint64_t. This unfortunately requires an API change. 2. The only pathological behavior happens if both arguments are INT64_MIN, to the best of my knowledge. In such a case, the implementation defined behavior is invoked in the sense that UINT64_MAX is interpreted as INT64_MIN, which any reasonable implementation will do. In any case, any usage where both arguments are INT64_MIN is a fuzzer anyway. 3. Alternatives of checking, etc require branching and lose performance for no concrete gain - no client cares about av_gcd's actual value when both args are INT64_MIN. Even if it did, on sane platforms (e.g all the ones FFmpeg cares about), it produces a correct gcd, namely INT64_MIN. Reviewed-by: Michael Niedermayer Signed-off-by: Ganesh Ajjanagadde --- libavutil/mathematics.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'libavutil') diff --git a/libavutil/mathematics.c b/libavutil/mathematics.c index 16e4eba5b9..fde460cc3e 100644 --- a/libavutil/mathematics.c +++ b/libavutil/mathematics.c @@ -52,7 +52,7 @@ int64_t av_gcd(int64_t a, int64_t b) { v -= u; v >>= ff_ctzll(v); } - return u << k; + return (uint64_t)u << k; } int64_t av_rescale_rnd(int64_t a, int64_t b, int64_t c, enum AVRounding rnd) -- cgit v1.2.3