Skip to content

cuda: Fix out-of-bounds source read in BGRX_TO_YUV420 kernel

Pascal Nowack requested to merge pnowack/gnome-remote-desktop:fix-out-of-bounds-read-in-bgrx-to-avc-cuda-kernel into master

Quoting the commit message here:

Compute operations on the GPU run in blocks. Each block consists of a
certain amount of threads.
Since these blocks are always a multiples of the warp size, additional
out-of-bounds checks are required when reading or writing memory on the
GPU.
For the NV12 buffer this is done in all cases. For the source data too,
except in one case, when reading the pixel x = 0, y = 1, in a 2x2
rectangle.
In that case, an invalid memory access can happen and all subsequent
CUDA operations will fail.

Fix this by adding the missing out-of-bounds check.
Edited by Pascal Nowack

Merge request reports