Fix undefined behavior in unsigned-to-signed conversion

The code assumed that `int x = - (unsigned) u` with 0 <= u < INT_MAX
sets `x` to the negative of u, but actually this calculates
(UINT_MAX - u) and then converts this value to int, which overflows.
Cast to int before applying the unary minus operator to guarantee the
desired behavior.
This commit is contained in:
Gilles Peskine 2018-10-12 19:19:12 +02:00
parent 9b430704d1
commit 4899247bf2

View file

@ -1578,7 +1578,7 @@ int mbedtls_rsa_rsaes_pkcs1_v15_decrypt( mbedtls_rsa_context *ctx,
* - OUTPUT_TOO_LARGE if the padding is good but the decrypted
* plaintext does not fit in the output buffer.
* - 0 if the padding is correct. */
ret = - if_int( bad, - MBEDTLS_ERR_RSA_INVALID_PADDING,
ret = - (int) if_int( bad, - MBEDTLS_ERR_RSA_INVALID_PADDING,
if_int( output_too_large, - MBEDTLS_ERR_RSA_OUTPUT_TOO_LARGE,
0 ) );