SIGN IN SIGN UP

bundle: Preallocate buffers for file contents. (#6818)

This commit adds logic to preallocate buffers when loading files from
both tarballs and on-disk bundle directories. The change results in
lower max RSS memory usage at runtime, and better garbage collector
performance, especially when at lower values of GOMAXPROCS.

For very large bundles (>1 GB in size), this change can lower startup
times for OPA by as much as a full second.

The performance analysis was different than for most changes-- heap
usage increased by about 10% during bundle loading, which made the
change look bad at first. Some of the effect appears to be from the
Go compiler no longer inlining as far up the call chain during bundle
loading (visible in the `pprof` graphs).

Running with `GODEBUG=gctrace=1` and varying GOMAXPROCS allowed seeing a
fuller picture of how performance changes from preallocation, which
results in much less garbage for the collector, and a noticeable speedup
in wall-clock time the GC burns during bundle loading.

Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
P
Philip Conrad committed
8e7172c8f8e0b03f654d66aeb95762d8220e46e8
Parent: 3b06458
Committed by GitHub <noreply@github.com> on 6/19/2024, 12:19:15 PM