Tobias Stoeckmann authored Oct 29, 2017 and Jehan committed Dec 22, 2017

It is possible to trigger a heap overflow while parsing FLI files. The
RLE decoder is vulnerable to out of boundary writes due to lack of
boundary checks.

The variable "framebuf" points to a memory area which was allocated
with fli_header->width * fli_header->height bytes. The RLE decoder
therefore must never write beyond that limit.

If an illegal frame is detected, the parser won't stop, which means
that the next valid sequence is properly parsed again. This should
allow GIMP to parse FLI files as good as possible even if they are
broken by an attacker or by accident.

While at it, I changed the variable xc to be of type size_t, because
the multiplication of width and height could overflow a 16 bit type.

Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>