We do a series of cleanups on the state object and we might have lost
our connection to the object graph and or the buffer itself. This handles
those situations gracefully (while still asserting correctness in devel
builds).

Fixes #1297