Fix an incorrect assert() in rtree that was added recently

by [f7ad73c49507c943].  Change it to testcase().

FossilOrigin-Name: 3aa0bf3a098ee9a62c79f266f94e4d290056d0f3e91237944893d6bbb686861f