Memory exhaustion (DoS) via unbounded file read in `EncodingDiscovery.detect_encoding()`

How it was found

Summary

EncodingDiscovery.detect_encoding() in translation_finder/discovery/base.py passes the raw file handle to charset_normalizer.from_fp(), which internally does from_bytes(fp.read()) — the entire file is loaded into memory, with no size cap. Every other sniffing path in this codebase is bounded at 1 MiB (FORMAT_SNIFF_MAX_BYTES in discovery/files.py); this one is not.

Result: a single oversized translation file (e.g. a 128 MiB cs.properties) makes a discover() run's peak RSS grow to ~3× the file size (~403 MiB), which deterministically OOM-kills a discovery worker on constrained containers. Repro and evidence below.

Vulnerability Details

• Weakness: CWE-770 (Allocation of Resources Without Limits or Throttling)
• Product: WeblateOrg/translation-finder 3.4.1 — also present on current main (e94e820)
• Environment: Python ≥ 3.11, Linux x86_64, charset-normalizer 3.4.3/3.5.1
• Reachable from: Weblate component discovery addon, Add Component flow, Transifex config import; runs after every VCS update

Root cause

translation_finder/discovery/base.py (detect_encoding, v3.4.1):

446  for path in chain(*matches):
447      if not isinstance(path, Path):
448          continue
449      with self.finder.open(path, "rb") as handle:
451          detection_result = from_fp(handle).best()   # <-- fp.read() whole file

charset_normalizer/api.py:

826  return from_bytes(
827      fp.read(),          # <-- no limit

Trigger conditions

A single matched file is enough, fully read in one call:

• Normal mode: any language-named file (locale/cs.properties) — and, as traced, plain filenames like data.properties are also fully read whenever a .properties component exists (detect_encoding() iterates all mask matches).
• Eager mode: every known-format file.
• Cost scales with file size (~3× RSS) and with the number of directories/files (from_fp is called once per match: 3 dirs with the same 16 MiB file = 48 MiB read).

Steps To Reproduce

Step 1. Setup

python3 -m venv tfenv && source tfenv/bin/activate
pip install "translation-finder==3.4.1"

Step 2. Repo with one large translation file (128 MiB)

mkdir -p /tmp/encpoc/locale
printf 'greeting=Hello\n' > /tmp/encpoc/locale/en.properties
python3 -c "open('/tmp/encpoc/locale/cs.properties','wb').write(b's=' * 67108864)"

Step 3. Watch memory

/usr/bin/time -v python3 -c "from translation_finder.api import discover; discover('/tmp/encpoc')" 2>&1 | grep "Maximum resident"

Expect ~400 MB peak for 128 MiB file (baseline ~12 MB)

Step 4. Worker-OOM proof (use a 32 MiB file for speed: b's=' * 16777216)

bash -c 'ulimit -v 90000; python3 -c "from translation_finder.api import discover; discover(\"/tmp/encpoc\")"' 2>&1 | tail -2

Expect: MemoryError at charset_normalizer/api.py (cap 130000 -> completes fine)

Remediation

Read a bounded sample, same as the other sniffers:

SNIFF_LIMIT = 1024 * 1024  # same as FORMAT_SNIFF_MAX_BYTES
with self.finder.open(path, "rb") as handle:
    sample = handle.read(SNIFF_LIMIT)
    detection_result = from_bytes(sample).best()

Or reuse _read_binary_sniff_content() / _is_sniff_content_over_limit() and skip oversized files entirely. The unbounded fp.read() in charset_normalizer.from_fp() is upstream behavior; the fix belongs in translation-finder.

Supporting Material

Reporter measurements (subprocess-isolated ru_maxrss, baseline ~11.8 MiB):

Independent re-verification (main@e94e820, charset-normalizer 3.5.1): 32 MiB file → peak 116.5 MiB (+103.3 MiB, 3.23×); instrumented from_fp: 1 call, exact file size read; cap test ulimit -v 90000 → MemoryError, 130000 → OK. Control: a 100 MiB cs.json through the capped path (_read_binary_sniff_content) adds only ~2 MiB RSS.

Attachments: poc_encoding_mem2.py, poc_oom_child.py, poc_aggregate_eager.py, poc_bytes_aggregate.py, results_measurements.txt.

Impact

Availability only. A 100 MiB blob (within GitHub's push limit) already costs ~300 MiB RSS per scan — enough to OOM a 512 MiB container; self-hosted forges allow much larger blobs. Triggered automatically on VCS updates, so no extra action needed beyond pushing to a watched repository. CVSS 3.1: AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H = 3.7 (admin/import flows) or 5.5 with PR:N where webhooks/PR-driven scans make it reachable by unprivileged contributors.