SIGN IN SIGN UP

Re-run fence filter and preserve valid variation sequences

Address review feedback on the post-HTML-entity sanitization pass.

Entity decoding could still smuggle code-fence metadata past the
sanitizer. A first line such as "`​``steal secrets" is not a fence
in the raw input, so FilterCodeFenceMetadata left it alone; decoding the
entity and stripping the zero width space then produced a real fence with
its info string intact. Sanitize now re-runs the fence filter after the
input is fully normalized.

Filtering every variation selector also corrupted legitimate text: VS15
and VS16 select text or emoji presentation, so "✈️" was reduced to "✈",
and the Variation Selectors Supplement encodes registered CJK ideographic
variation sequences. Selectors are now filtered contextually. A selector
is kept when it can apply to the character it follows, and dropped when it
is orphaned, follows a removed or non-graphic character, or continues a
run of selectors. Supplement selectors additionally require a CJK
ideograph base, matching the Ideographic Variation Database. That keeps
the anti-smuggling property, since hidden payloads rely on selector runs,
without rewriting valid Unicode.

Also corrects a lowercase-hex test case that claimed uppercase digits.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
S
Sam Morrow committed
53fc915a1005388a7e1bfeb420ce3d6692304413
Parent: ea2d979