(CVE-2023-32611) g_variant_byteswap() can take a long time with some non-normal inputs

Splitting out from #2121 (comment 832262)

With some non-normal GVariant inputs, such as those from https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=19777, the size of the serialised, normal form of the GVariant is huge (an order of magnitude larger than the input). That’s unavoidable. g_variant_byteswap() currently does its byteswapping on this serialised normal form, which means it can be exploited to do a DoS attack if handling malicious input.

Unfortunately, g_variant_byteswap() (along with g_variant_get_normal_form()) is often used as the first function to handle incoming untrusted GVariant input, so it must be robust against this kind of attack.

#2121 (closed) has made g_variant_get_normal_form() robust against this kind of attack. The same must be done with g_variant_byteswap().

The likely approach is to change g_variant_byteswap() to do its byteswapping on the tree form of the GVariant as returned by g_variant_get_normal_form(), rather than on a serialised version of that. That will preserve the pruning operations done by g_variant_get_normal_form() in response to #2121 (closed).