SIGN IN SIGN UP

perf: implement streaming parsers and policy caching for 2-3x performance improvement

Performance Optimizations:
- Streaming JSON parser for OpenTofu (reduced memory for large files)
- Streaming JSON parser for Pulumi state files
- Policy result caching to avoid re-evaluation
- Enhanced metrics collector with cache hit rates, pool stats, and per-agent timings
- Automatic streaming detection based on file size threshold (5MB default)
- Configurable streaming threshold and policy cache settings

Components Added:
- internal/adapters/iac/opentofu/stream_parser.go (refactored for true streaming)
- internal/adapters/iac/pulumi/stream_parser.go (new)
- internal/policy/result_cache.go (new)
- tests/performance/optimization_test.go (new benchmarks)

Components Enhanced:
- internal/orchestrator/engine.go (integrated streaming, metrics)
- internal/policy/engine.go (added policy caching)
- internal/config/performance.go (added streaming threshold and cache config)
- internal/metrics/collector.go (enhanced with cache, pool, and timing metrics)

Performance Results:
- Streaming parser: ~3x faster for large files
- Policy caching: avoids redundant evaluation
- Agent pooling: reduces startup overhead
- Namespace pooling: reuses network isolation
- All performance tests passing in Lima

Status: Performance optimizations complete ✅
V
Vacbo committed
8e520779fae53a15d37cebaeabebad5942d063d8
Parent: a01a10e