Avoid iterating over children when not placing for alignment

Caused a regression in the layout benchmark.

Fixes: 552164953
Test: RectsInFlowColumnBenchmark#layout_100
Change-Id: Ib766c1ad7656adc451c2125475613e9ea3002591