RDP: Add support for multitouch input events
This set of changes introduces multitouch input handling for the RDP backend. Touch (and (multi-)pen) input uses the state diagram in https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-rdpei/797930c0-d5ca-4da5-a801-97813d3b750e for each contact state.
Quoting the two most noteworthy commits here:
rdp: Add class for handling multitouch input events
Touch (and (multi-)pen) input is not part the RDP core protocol.
Instead, they are transmitted via the Input Virtual Channel Extension.
So add a class for this DVC and derive it from the GrdRdpDvc.
Compared to "normal input" (keyboard and mouse events), touch input in
RDP is state based. This means, that input events cannot just be
submitted to the RDP event queue. Instead, for each touch contact, the
resulting event is based on the state the touch contact is in, and the
contact flags, which determine the next state of the touch contact. This
state diagram is documented in 3.1.1.1 Touch Contact State Transitions
([MS-RDPEI]).
To not leak any [MS-RDPEI] specific types and internals to the RDP event
queue, handle these touch contact transitions inside this new DVC class.
Based on the type of the touch contact id (uint8), 256 simultaneous
touch contacts are possible. So add an array with 256 touch contexts to
this class.
Each touch context tracks the state of the automata in 3.1.1.1
([MS-RDPEI]) for the respective touch contact, a GrdTouchContact object,
and a flag, whether the contact should be ignored. The GrdTouchContact
object here is needed for the previously added API in GrdSession to
update a touch contact.
The ignore_contact flag is used to track whether a touch contact should
be ignored. This flag is used, when the touch position cannot be
transformed. Usually, that is the case, when the client sends a
position, that is outside of any (virtual) monitor here. The client does
not necessarily know, whether a touch contacts position cannot be
transformed and thus considers the contact to be valid. This is not a
problem, as the event can simply be ignored here. However, the follow-up
events depend on the contact state here. Thus, continue to handle the
contact flags of the event to transition the touch contact state in the
next state. When the original state ('Out of Range') is reached, reset
the ignore_contact flag.
When an input event is received, do not directly handle it. Instead,
push it to an internal queue, which is processed, just like the ei
source in GrdSession, on the main thread. This ensures, that no race
conditions happen, as ei touch contacts can only be acquired, when there
is an ei touch device.
Currently, only touch events are handled and not pen events, as libei is
missing the needed API for it. Also, in RDP, touch events can contain an
optional (rotated) contact rect and a value describing the pressure of
the touch contact. Discard these optional parameters, as there is no
respective libei API for them too.
rdp-dvc-input: Reset touch contexts when touch device is recreated
When the monitor layout of an active remote desktop session changes,
mutter recreates all ei regions and thus all ei touch devices too.
This makes sense, as the input regions changed. In this situation, all
active touch contacts become invalid and thus need to be destroyed.
On the side of the remote desktop client, the same seems to happen: The
internal state machine for each contact is reset to the 'Out of Range'
state.
The handling for this on the server side can actually be combined with
the recreation of the ei touch device. To do that, hook up the
previously added signals in GrdSession, that notify connected handlers,
that the ei touch device was destroyed. To handle a previously active
contact, simply release the GrdTouchContact and reset the state of the
[MS-RDPEI] specific contact automata.
During testing, a bug with Microsoft Windows App on the iPhone was
found: When a touch contact was held (touch-down) and the screen
rotated, then the client resets the automata states. However, for some
reason, it continues to send spurious touch-motion events with the same
position over and over again, even when the touch contact is not held
anymore or the finger moved to a different position while the touch
contact is held.
When performing a touch-down again, these spurious touch-motion events
are not sent anymore. The client here also sends the contact flags for a
touch-down event, meaning that the touch-motion events were not
intended.
To avoid unnecessary warnings, add a workaround for this client bug on
the server side: When the ei touch device is recreated and all related
touch contacts are reset, set the ignore_contact flag, when the contact
is not already in the 'Out of Range' state. In addition to that, reset
the ignore_contact flag, when handling the 'Out of Range' state with a
valid contact flag combination for that state.
This gets rid of a spam wave of these spurious touch-motion events,
while keeping protocol conform behaviour.
When an ei touch device is added, trigger the event source to ensure,
that events, that were received before the new ei touch device was
added, are actually handled. Use for this the previously added
'touch-device-added' signal.
Mostly tested with Microsoft Windows App for the iPhone. Also did a brief test on a workmates laptop (Lenovo Aura (Lunar Lake)) with mstsc on Win 11. Was a nice testing experience. With mstsc, MS Windows also shows (renders) touch input visuals, when doing some touch-motion.
Depends on !313 (merged) (commits included here)
Edited by Pascal Nowack