Clean up visible inlay hints that got removed from the cache (#7399)

Fixes another flack in the inlay hint cache.
Now that test does not fail after 500 iterations and seems to be stable
at last.

Release Notes:

- N/A