Track the RBNode/RBTree instead of keeping a TreeRowReference. This is a
whole lot faster and less error-prone.

Also, notify the accessible of removal of rows before actually removing
them, so we have a chance to clean up.