We currently do use the hashes collection, the RawTextComparator implementations reach inside and read it directly.  But I see your point.

I think removing it is going to be a bit more work than I'd like to squash into this refactoring change, so I'd like to actually defer it until after we have PatienceDiff in.  Some of the optimizations I'd like to do is to avoid computing the hashes for the ranges we don't care about (basically do it after common header/footer elimination... aka step 1 of patience diff).