language: Avoid UTF-16 false positive with embedded ASCII (#61250)
# Objective - Zed can hang (and eventually get force-killed) when opening certain binary files, because `analyze_byte_content`'s UTF-16 heuristic misclassifies them as UTF-16LE/BE text. - Reproduced with a real-world case: a ~92 MB OTBM game map file (the binary map format used by OpenTibia/Tibia servers), which interleaves short ASCII strings with small u16 length/type fields. Its byte pattern (mostly-zero high bytes, very few control characters) passed the existing check, so Zed read the entire file, decoded it as UTF-16, and opened it as an editable buffer with tens of millions of characters and effectively no line breaks — a pathological case for the text layout/renderer that hangs or crashes the app (most noticeably on Windows). ## Solution `is_plausible_utf16_text` in `crates/language/src/file_content.rs` previously only rejected the UTF-16 hypothesis when too many code units were control characters (> 2%). That's not sufficient on its own: binary formats that interleave short ASCII fragments with small numeric fields can have a very low control-character ratio while still not being real text — most of their "characters" land on stray symbol/high-byte values rather than letters, digits, or spaces. This PR adds a second, independent requirement: at least 30% of the analyzed code units must be letters, digits, or spaces (the bulk of any real UTF-16 text sample). Both conditions now have to hold for a byte sequence to be classified as UTF-16 text — otherwise it falls through to `ByteContent::Binary`, and file loading is rejected early, as intended for binary files, instead of decoding the whole file as garbled text. ## Testing - Added `test_length_prefixed_binary_not_misdetected_as_utf16le` in `crates/worktree/src/worktree.rs`, using a synthetic byte pattern that reproduces the same statistical shape as the real file (null high bytes, low control-character ratio, no word-like low bytes) — asserts it is now classified `Binary`. - Verified against the real 92 MB `.otbm` file that triggered the bug (not committed, since it's user data): before the fix it was classified `Utf16Le`, after the fix it's classified `Binary`. - Ran the full existing `analyze_byte_content` / `is_plausible_utf16_text` test suite (`cargo test -p worktree --lib tests::`) — all 7 tests pass, including the pre-existing positive UTF-16LE/UTF-16BE detection tests, so legitimate UTF-16 files are unaffected. - Built a full `--release` binary on Windows and confirmed opening the real file now shows "Binary files are not supported" immediately instead of hanging. ## Self-Review Checklist: - [x] I've reviewed my own diff for quality, security, and reliability - [x] Unsafe blocks (if any) have justifying comments — N/A, no unsafe code - [x] The content adheres to Zed's UI standards — N/A, no UI change - [x] Tests cover the new/changed behavior - [x] Performance impact has been considered and is acceptable — only affects classification of the first 1 KB of a file, negligible cost --- Release Notes: - Fixed: Zed no longer hangs when opening certain binary files (e.g. game asset/map formats) that were previously misdetected as UTF-16 text. --------- Co-authored-by: Kirill Bulatov <kirill@zed.dev>
S
Sarah Wesker committed
d2779c344350bcac5efabf840bbd31ba5b866ab4
Parent: 3b90a9b
Committed by GitHub <noreply@github.com>
on 8/8/2026, 1:32:13 PM