SIGN IN SIGN UP

language: Avoid UTF-16 false positive with embedded ASCII (#61250)

# Objective
- Zed can hang (and eventually get force-killed) when opening certain
binary files, because `analyze_byte_content`'s UTF-16 heuristic
misclassifies them as UTF-16LE/BE text.
- Reproduced with a real-world case: a ~92 MB OTBM game map file (the
binary map format used by OpenTibia/Tibia servers), which interleaves
short ASCII strings with small u16 length/type fields. Its byte pattern
(mostly-zero high bytes, very few control characters) passed the
existing check, so Zed read the entire file, decoded it as UTF-16, and
opened it as an editable buffer with tens of millions of characters and
effectively no line breaks — a pathological case for the text
layout/renderer that hangs or crashes the app (most noticeably on
Windows).

## Solution
`is_plausible_utf16_text` in `crates/language/src/file_content.rs`
previously only rejected the UTF-16 hypothesis when too many code units
were control characters (> 2%). That's not sufficient on its own: binary
formats that interleave short ASCII fragments with small numeric fields
can have a very low control-character ratio while still not being real
text — most of their "characters" land on stray symbol/high-byte values
rather than letters, digits, or spaces.

This PR adds a second, independent requirement: at least 30% of the
analyzed code units must be letters, digits, or spaces (the bulk of any
real UTF-16 text sample). Both conditions now have to hold for a byte
sequence to be classified as UTF-16 text — otherwise it falls through to
`ByteContent::Binary`, and file loading is rejected early, as intended
for binary files, instead of decoding the whole file as garbled text.

## Testing

- Added `test_length_prefixed_binary_not_misdetected_as_utf16le` in
`crates/worktree/src/worktree.rs`, using a synthetic byte pattern that
reproduces the same statistical shape as the real file (null high bytes,
low control-character ratio, no word-like low bytes) — asserts it is now
classified `Binary`.
- Verified against the real 92 MB `.otbm` file that triggered the bug
(not committed, since it's user data): before the fix it was classified
`Utf16Le`, after the fix it's classified `Binary`.
- Ran the full existing `analyze_byte_content` /
`is_plausible_utf16_text` test suite (`cargo test -p worktree --lib
tests::`) — all 7 tests pass, including the pre-existing positive
UTF-16LE/UTF-16BE detection tests, so legitimate UTF-16 files are
unaffected.
- Built a full `--release` binary on Windows and confirmed opening the
real file now shows "Binary files are not supported" immediately instead
of hanging.

## Self-Review Checklist:
- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments — N/A, no unsafe
code
- [x] The content adheres to Zed's UI standards — N/A, no UI change
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable — only
affects classification of the first 1 KB of a file, negligible cost

---

Release Notes:

- Fixed: Zed no longer hangs when opening certain binary files (e.g.
game asset/map formats) that were previously misdetected as UTF-16 text.

---------

Co-authored-by: Kirill Bulatov <kirill@zed.dev>
S
Sarah Wesker committed
d2779c344350bcac5efabf840bbd31ba5b866ab4
Parent: 3b90a9b
Committed by GitHub <noreply@github.com> on 8/8/2026, 1:32:13 PM