SIGN IN SIGN UP
oven-sh / bun UNCLAIMED

Incredibly fast JavaScript runtime, bundler, test runner, and package manager – all in one

0 0 150 Rust

Decode Buffer base64/base64url with simdutf's lenient accept-garbage mode (#31423)

### Motivation

`Buffer.from(str, "base64" | "base64url")` and `buf.write(str, "base64"
| "base64url")` decoded through `bun_base64::decode`, which calls
simdutf with the strict **standard** alphabet only (`base64_default`).
Any URL-safe character (`-`/`_`), non-alphabet byte, interior `=`, or
unusual padding made that call fail and the whole input was re-decoded
by a scalar fallback decoder — so `base64url` input containing its own
alphabet never got the SIMD path at all. Two-byte (UTF-16) strings also
decoded differently from Node.

simdutf (8.x, vendored via the WebKit headers) ships the lenient decode
modes that Node.js Buffer semantics map onto:
`base64_default_or_url_accept_garbage` — accept both alphabets in the
same input, skip whitespace and any other non-alphabet characters, stop
at the first `=`.

### What changed

- `src/simdutf_sys/bun-simdutf.cpp` / `simdutf.rs`: new
`simdutf__base64_decode_from_binary_lenient` wrapper around
`simdutf::base64_to_binary_safe(...,
base64_default_or_url_accept_garbage, loose)`.
- `src/base64/lib.rs`: new `bun_base64::decode_lenient()` (plus
`decode_lenient_len()`), implementing the Node Buffer semantics in one
simdutf call. The scalar fallback only remains for destinations smaller
than the worst-case decode (e.g. `buf.write` into a short buffer), where
simdutf's chunked strategy doesn't honor the stop-at-`=` rule on its own
— the input is cut at the first `=` up front so both strategies agree.
- `src/runtime/webcore/encoding.rs`:
- the Buffer `base64`/`base64url` arms of `construct_from_u8` and
`write_u8` use the lenient decode, and the scratch buffer is sized with
the worst-case bound (like Node) so the whole decode happens in a single
simdutf pass;
- two-byte (UTF-16) strings are now decoded from the **low byte of each
code unit** (the same narrowing the `latin1` encoding uses), matching
Node's lenient fallback decoder, instead of transcoding to UTF-8 and
skipping non-Latin-1 characters.
- Other `bun_base64::decode` callers (data URLs, ini, csrf, sourcemaps,
postgres, images) are unchanged.

### Behavior

For one-byte (Latin-1) strings there is no behavior change — the lenient
mode matches what the previous simdutf-then-scalar-fallback combination
produced.

For two-byte (UTF-16) strings, decoding now matches Node.js exactly:
each code unit is narrowed to its low byte before the lenient decode, so
e.g. `U+D83D` (the first unit of 😀) behaves like `=` and stops decoding,
and `U+1234` behaves like `4` and contributes data. Previously Bun
skipped such characters, producing different output than Node:

```js
Buffer.from("QUJD\u{1F600}REVG", "base64").toString("latin1")
// Node and Bun with this PR: "ABC"     Bun before: "ABCDEF"

Buffer.from("\u1234QUJDREVG", "base64")
// Node and Bun with this PR: <e1 05 09 0d 11 15>     Bun before: "ABCDEF"
```

Verification:

- A 244,000-line randomized differential corpus
(garbage/padding/alphabet/whitespace/non-Latin-1 permutations through
`Buffer.from` and `buf.write` with random destination sizes, both
encodings) now produces **byte-for-byte identical output to Node v24**;
before this PR the same corpus had ~34,000 differing lines (all
two-byte-string cases).
- A 1,236-input structured corpus (Node's buffer test edge cases plus
padding/garbage/alphabet permutations, every ASCII byte, long inputs) is
byte-for-byte identical to Bun 1.4.0 for one-byte strings.
- `test/js/node/buffer.test.js` (488 tests) and the vendored Node suites
`test-buffer-alloc.js`, `test-buffer-write.js`,
`test-buffer-bytelength.js`, `test-buffer-from.js`,
`test-buffer-tostring.js`, `test-buffer-fill.js` pass with the debug
build.
- New tests pin the lenient semantics (garbage skipping, mixed
alphabets, stop-at-`=`, whitespace, undersized `write()`), and the new
two-byte tests fail on Bun without this change and pass with it (and
pass on Node).

### Performance

Same machine, sequential runs, identical inputs (benchmark added as
`bench/snippets/buffer-base64-decode.mjs`); details in the benchmark
comment below.

| benchmark | this PR | main | Bun 1.3.14 | Node 24.3 |
|---|---|---|---|---|
| `Buffer.from` 1 MiB base64 (clean) | **141 µs** | 176 µs | 166 µs |
208 µs |
| `Buffer.from` 1 MiB base64url (clean) | **138 µs** | 2.30 ms | 3.88 ms
| 207 µs |
| `Buffer.from` 1 MiB base64 + CRLF wrapping | **161 µs** | 205 µs | 179
µs | 250 µs |
| `Buffer.from` 1 MiB URL alphabet as `"base64"` | **169 µs** | 2.30 ms
| 3.83 ms | 1.06 ms |
| `Buffer.from` 64 KiB base64, 1% garbage | **10.2 µs** | 160 µs | 248
µs | 71.8 µs |
| `Buffer.from` 512-byte base64url | **~375 ns** | 1.52 µs | 2.28 µs |
~350–385 ns |
| `buf.write` 1 MiB base64 | ~74 µs | ~73 µs | ~72 µs | ~72 µs |

Clean input decodes with simdutf's strict per-alphabet kernel (same
first attempt Node makes); the lenient accept-garbage kernel only runs
when the input needs it; `Buffer.from` no longer zero-fills the output
buffer before decoding.

### Notes

- `base64url` input and inputs containing whitespace/garbage now decode
through simdutf's SIMD kernels instead of the scalar fallback.
- The two-byte narrowing applies to `Buffer.from`, `buf.write`, and
`buf.fill` with `base64`/`base64url` encodings.
R
robobun committed
2148214d40fc9f844f61ae5f8575d6d94a9d038c
Parent: 18b7c95
Committed by GitHub <noreply@github.com> on 5/26/2026, 10:08:25 PM