SIGN IN SIGN UP

Serialization: much faster `(de)serialize_cycle` by refactoring the lookup tables (#61766)

closes https://github.com/JuliaLang/julia/issues/44175

`serialize` contains an optimization to deduplicate identical objects
both for performance benefit and to retain aliasing relationships during
round trips.

conceptually, it does this by: for each value that is eligible for
`serialize_cycle` (in practice, this mostly resolves to `ismutable`),
maintain `counter` for the index at which we encountered that object,
and save the `objectid` for future lookups. Then, if we see the same
`objectid` later, serialize a `BACKREF` to the appropriate `counter`.

On the deser side, when we see a `BACKREF`, access the object at the
index of the associated `counter` rather than making a new object.

however, it uses an `IdDict` for both purposes. this PR squeezes out a
lot more performance by using a `Dict` for the serialization lookup and
a `Vector` for the deser.

questions for the reviewers mostly concern compatibility. this does
change the fields of `Serializer` which is (unfortunately) a public
type. but I'm not sure if the fields themselves are considered public? I
have headed off in advance impacts to, e.g.
[`Distributed.ClusterSerializer`](https://github.com/JuliaLang/Distributed.jl/blob/fde987fbb05c65bb6bd183396137383931107059/src/clusterserialize.jl#L10-L14)
which makes assumptions about the way that `Serialization.serialize`
makes use of a `.table` field on `AbstractSerializer`, by factoring out
these field accesses into getter/setter functions and then specializing
on `Serializer`. Is this an appropriate compromise? or should we just
patch `Distributed` to not rely on this assumption? or is the change to
`Serialization` considered wholesale breaking. there was a similar
discussion in https://github.com/JuliaLang/julia/pull/43554

here are performance numbers on a few workloads:

on the motivating MWE from
https://github.com/JuliaLang/julia/issues/44175:
<details>
<summary> setup</summary>

```julia
mutable struct MinimalNode
    childvalues::Union{NTuple{4, Int8}, NTuple{3, Int8}, NTuple{2, Int8}, NTuple{1, Int8}, Tuple{}};
    childnodes::Union{NTuple{4, MinimalNode}, NTuple{3, MinimalNode}, NTuple{2, MinimalNode}, NTuple{1, MinimalNode}, Tuple{}};
end

function addbranches!(mn::MinimalNode, aleft::Int64, bleft::Int64, parity::Int64, depth = 0)

    (depth == 12) && (return nothing);

    if parity == 1
        (aleft <= 1) && (return nothing);
        mn.childvalues = Tuple(1:aleft);
        mn.childnodes = Tuple([MinimalNode((), ()) for ii in 1:aleft]);
        aleft -= 1;
    else
        (bleft <= 1) && (return nothing);
        mn.childvalues = Tuple(1:bleft);
        mn.childnodes = Tuple([MinimalNode((), ()) for ii in 1:bleft]);
        bleft -= 1;
    end

    addbranches!.(mn.childnodes, aleft, bleft, 3-parity, depth + 1);
    
end

function buildfaketree()
    root = MinimalNode((), ());
    addbranches!(root, 4, 4, 1, 0);
    return root;
end


using Serialization;
MN = [buildfaketree() for ii in 1:10000];
@time serialize("test.jls", MN);
35 seconds
```
</details>

PR:
```julia
julia> @time serialize("test.jls", MN);
  6.419340 seconds (15.78 M allocations: 784.602 MiB, 17.18% gc time)

julia> @time deserialize("test.jls");
  4.382877 seconds (20.79 M allocations: 817.876 MiB, 32.13% gc time)
```

master:
```julia
julia> @time serialize("test_master.jls", MN);
 17.084042 seconds (26.55 M allocations: 917.083 MiB, 5.38% gc time)

julia> @time deserialize("test_master.jls");
 20.479949 seconds (31.56 M allocations: 1.204 GiB, 15.45% gc time)
```

but we also improve something as simple as `Vector{String}`

setup:
```julia
julia> using Random

julia> v = map((_)->randstring(10), 1:1000000);
```

PR:
```julia
julia> @time serialize("string.jls", v);
  0.240658 seconds (37 allocations: 59.501 MiB, 9.13% gc time)

julia> @time deserialize("string.jls");
  0.084324 seconds (1.00 M allocations: 64.112 MiB)
```

master:
```julia
julia> @time serialize("string.jls", v);
  0.592193 seconds (999.51 k allocations: 43.252 MiB, 11.42% gc time)

julia> @time deserialize("string.jls");
  0.371651 seconds (2.00 M allocations: 81.410 MiB, 15.02% gc time)
```

assisted as always by Opus 4.7
A
Andy Dienes committed
a6ef87570a46ea62507cfedb570569aabcf7a3d1
Parent: ba8af07
Committed by GitHub <noreply@github.com> on 5/14/2026, 12:43:23 AM