Tobias Stoeckmann authored Oct 31, 2017 and Michael Natterer committed Jan 01, 2018

The C language only promotes data values up to (un)signed int,
which is 32 bit, if no larger data type is used within the
calculation. Having a multiplication of two gint variables,
even if the expected target variable is of type gsize (64 bit),
leads to a possible integer overflow.

This bug can be triggered in gimp_temp_buf_new, which is used
to allocate memory for given supplied dimensions and bytes per
pixel. If triggered, less memory than needed is allocated and
therefore allows out of boundary accesses, either resulting in
possible code execution or information leakage.

While at it, make sure that the supplied format can actually be
resolved to a bytes per pixel value. If not, return NULL.

Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>