There will be a lot of these updates when the buffer is initially loading.
So just keep delaying a bit until we slow down on invalidation and can
actually land at our target location.

The amount of time this can take varies widely with whatever the idle
handler is doing. I've seen it be quick, and very slow.

Like always, going to require more hacks in gtktextview.c