You should use ObjectId, to prevent a MutableObjectId from being stored and changed behind your back.

Also, due to the need to support AnyObjectId during openObject2 below, and the oddity of how RevObject gets used, I would actually use a ObjectIdSubclassMap<ObjectId> here instead of the HashSet.  Its the same type we use to support efficient traversal in a RevWalk, but probably will use a lot less memory, and knows how to work around the RevObject hashCode/equals braindamage.