Fix some errors in the guttman versions (disabled by default) of the algorithms in rtree.c.

FossilOrigin-Name: 64bad00b4f6fbbc3e5e75966f9c3959ad3d542ef