OpenSSL backend: X509 peer certificate not freed in ossl_get_channel_binding causes per-request memory leak (DoS risk for long-lived clients)

How it was found

Summary:

In curl’s OpenSSL backend, ossl_get_channel_binding retains a new reference to the server’s X509 certificate via SSL_get1_peer_certificate and never releases it. When Negotiate (SPNEGO) over TLS is in use, this path is invoked and leaks one X509 object per trigger. Over many requests in a long‑running libcurl client, this leads to unbounded memory growth and potential denial‑of‑service.

This occurs only with the OpenSSL backend (>= 1.1.0 gate present), when curl/libcurl is built with GSSAPI (HAVE_GSSAPI), over HTTPS when a Negotiate challenge is processed.

Notes:

• The Negotiate challenge over HTTPS is sufficient to trigger channel binding retrieval; a functional Kerberos setup is not required for this leak to manifest.
• Each trigger leaks one X509 reference.

Supporting Material/References:

• Code location showing missing X509_free on all paths:

  cert = SSL_get1_peer_certificate(octx->ssl);
  if(!cert) {
    /* No server certificate, don't do channel binding */
    return CURLE_OK;
  }

  if(!OBJ_find_sigid_algs(X509_get_signature_nid(cert), &algo_nid, NULL)) {
    failf(data,
          "Unable to find digest NID for certificate signature algorithm");
    return CURLE_SSL_INVALIDCERTSTATUS;
  }

  /* https://datatracker.ietf.org/doc/html/rfc5929#section-4.1 */
  if(algo_nid == NID_md5 || algo_nid == NID_sha1) {
    algo_type = EVP_sha256();
  }
  else {
    algo_type = EVP_get_digestbynid(algo_nid);
    if(!algo_type) {
      algo_name = OBJ_nid2sn(algo_nid);
      failf(data, "Could not find digest algorithm %s (NID %d)",
            algo_name ? algo_name : "(null)", algo_nid);
      return CURLE_SSL_INVALIDCERTSTATUS;
    }
  }

  if(!X509_digest(cert, algo_type, buf, &length)) {
    failf(data, "X509_digest() failed");
    return CURLE_SSL_INVALIDCERTSTATUS;
  }

  /* Append "tls-server-end-point:" */
  if(curlx_dyn_addn(binding, prefix, sizeof(prefix) - 1) != CURLE_OK)
    return CURLE_OUT_OF_MEMORY;
  /* Append digest */
  if(curlx_dyn_addn(binding, buf, length))
    return CURLE_OUT_OF_MEMORY;

  return CURLE_OK;


Statement on AI usage: AI was used in parts of detection, triage, fix generation and reporting writing in combination with classical tooling and manual human input.

## Impact

## Impact

A remote HTTPS server that advertises `WWW-Authenticate: Negotiate` can cause a libcurl client (built with OpenSSL and GSSAPI) to leak one X509 certificate reference per authentication attempt when TLS channel binding is retrieved.

Over many requests in a long-lived process, this leads to unbounded memory growth and a denial-of-service condition due to resource exhaustion.

The leak occurs because `ossl_get_channel_binding` calls `SSL_get1_peer_certificate` (which increments the X509 reference count) but never calls `X509_free` on any return path—success or error.

### Constraints:
- Requires libcurl built with OpenSSL (>=1.1.0) and GSSAPI support
- Only affects clients using Negotiate authentication over HTTPS
- Linear leak rate (not amplified), but unbounded over time

Severity: **Low** (limited scope, requires specific build configuration and authentication method, but genuine DoS risk for affected deployments)