We could wrap it in an unmodifiableSet, but I don't care enough to bother.

If a caller finds out they can change this set, and does, and we later start caching it, you can bet that then we'll use an unmodifiableSet... and they will crash, and it wasn't an API breakage, because we told you its not modifiable.