Health check errors silently dropped when channel buffer full

How it was found

Component: pkg/plugin/plugin.go:153-156, pkg/plugin/plugin_v2.go:156-158 Affected Version: aws-encryption-provider @ 4341c70 (all versions) Found by: Source audit TLP: TLP:Amber

Summary

When KMS operations fail, the error is sent to a buffered channel (healthCheckErrc, size 100) via a non-blocking send. When the buffer is full, errors are silently dropped. Under sustained KMS failure, the health check goroutine's error state becomes stale, and /healthz may report healthy when KMS is actually down.

Vulnerable Code

// pkg/plugin/plugin.go:152-156 (also plugin_v2.go:155-159)
if err != nil {
    select {
    case p.healthCheck.healthCheckErrc <- err:
    default:
        // ERROR SILENTLY DROPPED
    }

The channel is created with a fixed buffer:

// pkg/plugin/shared_health_check.go:34
healthCheckErrc: make(chan error, errcBufSize),  // errcBufSize = 100

Root Cause

The non-blocking send pattern is intentional — it prevents Encrypt/Decrypt from blocking when the error channel is full. However, the silent drop means the SharedHealthCheck.Start() goroutine (which reads from this channel and calls RecordErr) may miss recent errors. If errors arrive faster than the goroutine consumes them (which happens under sustained failure), health state becomes stale.

Suggested Fix

Replace channel-based error propagation with direct RecordErr calls (already used in Health() method), or log when errors are dropped:

select {
case p.healthCheck.healthCheckErrc <- err:
default:
    p.healthCheck.RecordErr(err) // update state directly when channel full
}

Platform

All platforms. Source audit finding.

Impact

Impact

• Under sustained KMS failure (>100 errors queued), new errors are dropped
• Health check timestamp (lastTs) isn't updated from dropped errors
• /healthz reports stale status, potentially masking ongoing outages
• Kubernetes may not detect the provider is non-functional
• The existing TestHealthManyRequests test verifies non-blocking behavior but doesn't verify state correctness after drops