Remove unused `color` crate (#12860)

This PR removes the `color` crate, as it was not used anywhere.

We had added this experimentally, but right now its existence is just a
source of confusion.

Release Notes:

- N/A