[GTK3, GTK4] Animation based on continuous pointing device input is jittery

TLDR

Scrolling a window by using a touchpad is jittery even when we move the fingers at a constant speed. The somewhat counterintuitive notion here is that the scroll is jittery even though the input event rate is higher than the screen refresh rate. In fact even if the whole stack was fully optimized this issue would still exist.

The root cause

The root cause of the problem is that the input events are not synchronized to the screen refresh - they operate at different rates. As an example, the Apple external "Magic" trackpads emit input events at a rate of about 90Hz. A standard display operates at 60Hz. In this setup we get 2 input events in the first display frame, 1 input event in the second display frame, 2 in the third, 1 in the forth etc.

If, for example, each scroll event results in a movement of 4 pixels, we'll get 8px on the first frame (2 events of 4px each), 4px on the second frame (1 event of 4px), 8px on the third frame, 4px on the forth etc. So the window is scrolled by a wildly different number of pixels in each frame, even though the fingers moved on the touchpad at a constant speed.

The issue

In many setups, implicit animations caused by continuous input from a pointing device is not as smooth as it could be, and sometimes visibly jittery. This includes moving the mouse cursor, dragging a window, scrolling etc.

Affected touchpads

I personally tested the Apple external Magic Trackpads, both v1 and v2, which operate at about 88Hz.
I also tested the internal touchpad of the Dell Inspiron 3493. The latter has a "KIOXIA Corporation" touchpad which operates at about 143Hz according to the evhz utility.
According to !1117 (comment 635659) the issue also affects the Synaptics TM3276-022 touchpad found on the Lenovo T480. This touchpad operates at about 135Hz.

Considering the variety of the touchpad models listed above, I would guess that many setups are affected.

This issue is a sibling of #2573.

Steps to reproduce

I am using a first-generation Apple external 'Trackpad'. This was done on a desktop computer with an Intel 'Skylake' CPU and integrated GPU. The same issue was observed when using a discreet AMD GPU. The issue can manifest on any setup - it is actually a matter of input rate vs display rate, as explained in the TLDR above.

Run gtk3-demo (or gtk4-demo)
Click on the source tab of any demo to view its source code
Scroll using a 2-finger touchpad scroll

Current behavior

Scrolling is jittery.

Expected outcome

Scrolling should be smooth.

Version information

The desktop environment is GNOME Shell (v3.28.4) Wayland session on Ubuntu 18.04. The same issue exists on GTK4 (built from source) as well as the latest version of GTK v3.24 when building from source.

Additional information

Following is a recording of the jittery scroll. This is the current behavior: jitter_window

This is a recording of a smooth scroll session, taken after applying a proof-of-concept fix: smooth_window

More detailed explanation

The root cause of the problem is that the input events are not synchronized to the screen refresh - they operate at different rates. For example my external Apple 'Trackpad' generates around 90 input events per second, while the screen refreshes 60 times per second. So we get an input event about every 11 milliseconds while we get a screen refresh about every 17 milliseconds.

The following log snippet was generated with GDK_DEBUG=events gtk3-demo while trying to scroll at a constant speed. I removed unrelated log messages from the snippet. The "scroll" messages are printed in pointer_handle_axis() in gdk/wayland/gdkdevice-wayland.c while the "frame" messages are printed in frame_callback() in gdk/wayland/gdkwindow-wayland.c:

...
Gdk-Message: 10:36:46.723: scroll, axis horizontal, value 0.306641, seat 0x5555e5a0e030
Gdk-Message: 10:36:46.725: scroll, axis horizontal, value 0.311719, seat 0x5555e5a0e030
Gdk-Message: 10:36:46.728: frame 0x5555e5da70a0
Gdk-Message: 10:36:46.736: scroll, axis horizontal, value 0.322266, seat 0x5555e5a0e030
Gdk-Message: 10:36:46.745: frame 0x5555e5da70a0
Gdk-Message: 10:36:46.750: scroll, axis horizontal, value 0.332422, seat 0x5555e5a0e030
Gdk-Message: 10:36:46.759: scroll, axis horizontal, value 0.322266, seat 0x5555e5a0e030
Gdk-Message: 10:36:46.761: frame 0x5555e5da70a0
Gdk-Message: 10:36:46.770: scroll, axis horizontal, value 0.322266, seat 0x5555e5a0e030
Gdk-Message: 10:36:46.778: frame 0x5555e5da70a0
Gdk-Message: 10:36:46.783: scroll, axis horizontal, value 0.311719, seat 0x5555e5a0e030
Gdk-Message: 10:36:46.793: scroll, axis horizontal, value 0.316797, seat 0x5555e5a0e030
Gdk-Message: 10:36:46.795: frame 0x5555e5da70a0
Gdk-Message: 10:36:46.804: scroll, axis horizontal, value 0.311719, seat 0x5555e5a0e030
Gdk-Message: 10:36:46.811: frame 0x5555e5da70a0
...

The following diagram depicts the event timeline:

Time (ms) 0               17               34               51               68               85              102
VSync     |----------------|----------------|----------------|----------------|----------------|----------------|
Input         I          I          I          I          I          I          I          I          I

As can be seen in the log and in the timeline above, when scrolling on a standard 60FPS display we get 2 scroll events on the first frame, 1 scroll event on the second frame, 2 on the third frame, 1 on the forth and so on.

I tried to scroll at a constant speed. If, for example, each scroll event results in a movement of 4 pixels, we'll get 8px on the first frame (2 events of 4px each), 4px on the second frame (1 event of 4px), 8px on the third frame, 4px on the forth etc. So the window is scrolled by a wildly different number of pixels in each frame even though the fingers moved on the touchpad at a constant speed.

GDK Sub issues and related MRs

This list is maintained here in order to track the status of relevant GDK events.

#2166 (fixed by !1117) Precise scroll events are not synchronized to screen refresh
#2547 (fixed by !1562) Touchpad swipe/pinch events are not synchronized to screen refresh
#2548 (fixed by !1581) Touchscreen events are not synchronized to screen refresh

Origin

This issue originated from a discussion on a Firefox bug. I opened this bug report because we came to the conclusion that this is in fact a GTK issue. I re-posted the original bug report with the Firefox videos in the comments.

There is a corresponding gnome-shell#1471 issue, however since this is a system-wide problem I started a discussion on discourse here.

Edited Aug 08, 2020 by Yariv