Re-run fence filter and preserve valid variation sequences
Address review feedback on the post-HTML-entity sanitization pass. Entity decoding could still smuggle code-fence metadata past the sanitizer. A first line such as "`​``steal secrets" is not a fence in the raw input, so FilterCodeFenceMetadata left it alone; decoding the entity and stripping the zero width space then produced a real fence with its info string intact. Sanitize now re-runs the fence filter after the input is fully normalized. Filtering every variation selector also corrupted legitimate text: VS15 and VS16 select text or emoji presentation, so "✈️" was reduced to "✈", and the Variation Selectors Supplement encodes registered CJK ideographic variation sequences. Selectors are now filtered contextually. A selector is kept when it can apply to the character it follows, and dropped when it is orphaned, follows a removed or non-graphic character, or continues a run of selectors. Supplement selectors additionally require a CJK ideograph base, matching the Ideographic Variation Database. That keeps the anti-smuggling property, since hidden payloads rely on selector runs, without rewriting valid Unicode. Also corrects a lowercase-hex test case that claimed uppercase digits. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
S
Sam Morrow committed
53fc915a1005388a7e1bfeb420ce3d6692304413
Parent: ea2d979