window-actor-x11: Check array bounds before accessing array
scan_visible_region() scans through each value of a uint8_t array and checks whether that value is 255. Right now it always checks one value too much though, resulting in a buffer overflow. Fix that by checking the array bounds before actually accessing the array.
Found by running gnome-shell with address sanitizer and starting GIMP.
ASAN error message
Feb 17 12:12:05 suagaze org.gnome.Shell.desktop[103736]: =================================================================
Feb 17 12:12:05 suagaze org.gnome.Shell.desktop[103736]: ==103736==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7fb72212a028 at pc 0x7fb7508d264a bp 0x7ffe8404d280 sp 0x7ffe8404d278
Feb 17 12:12:05 suagaze org.gnome.Shell.desktop[103736]: READ of size 1 at 0x7fb72212a028 thread T0
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #0 0x7fb7508d2649 in scan_visible_region ../../../../jhbuild/checkout/mutter/src/compositor/meta-window-actor-x11.c:726
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #1 0x7fb7508d2649 in build_and_scan_frame_mask ../../../../jhbuild/checkout/mutter/src/compositor/meta-window-actor-x11.c:852
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #2 0x7fb7508d2649 in update_shape_region ../../../../jhbuild/checkout/mutter/src/compositor/meta-window-actor-x11.c:925
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #3 0x7fb7508d3287 in update_regions ../../../../jhbuild/checkout/mutter/src/compositor/meta-window-actor-x11.c:1066
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #4 0x7fb7508d3287 in update_regions ../../../../jhbuild/checkout/mutter/src/compositor/meta-window-actor-x11.c:1061
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #5 0x7fb7508d3287 in meta_window_actor_x11_update_regions ../../../../jhbuild/checkout/mutter/src/compositor/meta-window-actor-x11.c:1374
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #6 0x7fb750994dcb in meta_wayland_surface_role_apply_state ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-surface.c:1959
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #7 0x7fb750994dcb in meta_wayland_surface_apply_state ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-surface.c:869
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #8 0x7fb7509a5d46 in meta_wayland_transaction_apply ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-transaction.c:201
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #9 0x7fb7509a5d46 in meta_wayland_transaction_maybe_apply_one ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-transaction.c:258
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #10 0x7fb7509a5d46 in meta_wayland_transaction_maybe_apply ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-transaction.c:268
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #11 0x7fb7509a6927 in meta_wayland_transaction_commit ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-transaction.c:374
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #12 0x7fb7509900ab in meta_wayland_surface_commit ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-surface.c:1012
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #13 0x7fb7509900ab in wl_surface_commit ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-surface.c:1169
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #14 0x7fb752303a05 in ffi_call_unix64 (/lib64/libffi.so.8+0x7a05)
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #15 0x7fb75230049c in ffi_call_int.lto_priv.0 (/lib64/libffi.so.8+0x449c)
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #16 0x7fb752303082 in ffi_call (/lib64/libffi.so.8+0x7082)
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #17 0x7fb752358425 in wl_closure_invoke ../../../../jhbuild/checkout/wayland/src/connection.c:1025
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #18 0x7fb7523522c9 in wl_client_connection_data ../../../../jhbuild/checkout/wayland/src/wayland-server.c:438
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #19 0x7fb752355ea1 in wl_event_loop_dispatch ../../../../jhbuild/checkout/wayland/src/event-loop.c:1027
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #20 0x7fb750963e6a in wayland_event_source_dispatch ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland.c:114
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #21 0x7fb751f12120 in g_main_dispatch ../../../../jhbuild/checkout/glib/glib/gmain.c:3444
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #22 0x7fb751f12120 in g_main_context_dispatch ../../../../jhbuild/checkout/glib/glib/gmain.c:4162
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #23 0x7fb751f123b7 in g_main_context_iterate ../../../../jhbuild/checkout/glib/glib/gmain.c:4238
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #24 0x7fb751f12796 in g_main_loop_run ../../../../jhbuild/checkout/glib/glib/gmain.c:4438
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #25 0x7fb75084afef in meta_context_run_main_loop ../../../../jhbuild/checkout/mutter/src/core/meta-context.c:482
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #26 0x403bba in main ../../../../jhbuild/checkout/gnome-shell/src/main.c:668
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #27 0x7fb75044a50f in __libc_start_call_main (/lib64/libc.so.6+0x2750f)
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #28 0x7fb75044a5c8 in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x275c8)
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #29 0x4040a4 in _start (/home/tester/jhbuild/install/bin/gnome-shell+0x4040a4)
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: 0x7fb72212a028 is located 0 bytes to the right of 1275944-byte region [0x7fb721ff2800,0x7fb72212a028)
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: allocated by thread T0 here:
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #0 0x7fb7524ba097 in calloc (/lib64/libasan.so.8+0xba097)
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #1 0x7fb751f18e07 in g_malloc0 ../../../../jhbuild/checkout/glib/glib/gmem.c:163
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #2 0x7fb7508d1cd1 in build_and_scan_frame_mask ../../../../jhbuild/checkout/mutter/src/compositor/meta-window-actor-x11.c:806
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #3 0x7fb7508d1cd1 in update_shape_region ../../../../jhbuild/checkout/mutter/src/compositor/meta-window-actor-x11.c:925
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #4 0x7fb7508d3287 in update_regions ../../../../jhbuild/checkout/mutter/src/compositor/meta-window-actor-x11.c:1066
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #5 0x7fb7508d3287 in update_regions ../../../../jhbuild/checkout/mutter/src/compositor/meta-window-actor-x11.c:1061
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #6 0x7fb7508d3287 in meta_window_actor_x11_update_regions ../../../../jhbuild/checkout/mutter/src/compositor/meta-window-actor-x11.c:1374
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #7 0x7fb750994dcb in meta_wayland_surface_role_apply_state ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-surface.c:1959
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #8 0x7fb750994dcb in meta_wayland_surface_apply_state ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-surface.c:869
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #9 0x7fb7509a5d46 in meta_wayland_transaction_apply ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-transaction.c:201
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #10 0x7fb7509a5d46 in meta_wayland_transaction_maybe_apply_one ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-transaction.c:258
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #11 0x7fb7509a5d46 in meta_wayland_transaction_maybe_apply ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-transaction.c:268
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #12 0x7fb7509a6927 in meta_wayland_transaction_commit ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-transaction.c:374
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #13 0x7fb7509900ab in meta_wayland_surface_commit ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-surface.c:1012
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #14 0x7fb7509900ab in wl_surface_commit ../../../../jhbuild/checkout/mutter/src/wayland/meta-wayland-surface.c:1169
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: #15 0x7fb752303a05 in ffi_call_unix64 (/lib64/libffi.so.8+0x7a05)
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: SUMMARY: AddressSanitizer: heap-buffer-overflow ../../../../jhbuild/checkout/mutter/src/compositor/meta-window-actor-x11.c:726 in scan_visible_region
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Shadow bytes around the buggy address:
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: 0x0ff76441d3b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: 0x0ff76441d3c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: 0x0ff76441d3d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: 0x0ff76441d3e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: 0x0ff76441d3f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: =>0x0ff76441d400: 00 00 00 00 00[fa]fa fa fa fa fa fa fa fa fa fa
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: 0x0ff76441d410: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: 0x0ff76441d420: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: 0x0ff76441d430: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: 0x0ff76441d440: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: 0x0ff76441d450: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Shadow byte legend (one shadow byte represents 8 application bytes):
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Addressable: 00
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Partially addressable: 01 02 03 04 05 06 07
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Heap left redzone: fa
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Freed heap region: fd
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Stack left redzone: f1
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Stack mid redzone: f2
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Stack right redzone: f3
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Stack after return: f5
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Stack use after scope: f8
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Global redzone: f9
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Global init order: f6
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Poisoned by user: f7
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Container overflow: fc
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Array cookie: ac
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Intra object redzone: bb
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: ASan internal: fe
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Left alloca redzone: ca
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: Right alloca redzone: cb
Feb 17 12:12:06 suagaze org.gnome.Shell.desktop[103736]: ==103736==ABORTINGEdited by Jonas Dreßler