gegl-sampler sometimes crashes for absurd coordinates.
Submitted by Simon Budig
Link to original bug (#772119)
Description
Created attachment 336440 Demo program provoking the crash on 64 bit machines.
Under some circumstances the range check in gegl/buffer/gegl-sampler.h fails to recognize, that it needs to fetch data.
This happens when x and y coordinate are close to 2^31 (at least on my 64 bit machine).
Attached is a test program. For me this crashes on the 3rd gegl_sampler_get().
I have added some debug output to gegl that prints out the ROIs around the range checks. This is the output from a run with this debug output enabled:
This is the output from a gdb run with a LINEAR sampler:
(gdb) run
Starting program: /home/simon/src/unstable/gegl/tests/simple/.libs/test-sampler
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
need (2147483647, 2147483647; 3, 3), have (0, 0; 0, 0)
0.000000,0.000000,0.000000,0.000000
need (2147483647, 85707687; 3, 3), have (0, 0; 0, 0)
fetching (2147483645, 85707685; 9, 9)
0.000000,0.000000,0.000000,0.000000
need (2147483647, 2147483647; 3, 3), have (2147483645, 85707685; 9, 9)
Program received signal SIGSEGV, Segmentation fault.
gegl_sampler_linear_get (self=0x6c20a0 [GeglSamplerLinear],
absolute_x=<optimized out>, absolute_y=<optimized out>,
scale=<optimized out>, output=0x7fffffffdf60, repeat_mode=GEGL_ABYSS_NONE)
at gegl-sampler-linear.c:208
208 const gfloat bot_rite_3 = *in_bptr;
(gdb) bt
#0 0x00007ffff7b6b92b in gegl_sampler_linear_get (self=0x6c20a0 [GeglSamplerLinear], absolute_x=<optimized out>, absolute_y=<optimized out>, scale=<optimized out>, output=0x7fffffffdf60, repeat_mode=GEGL_ABYSS_NONE)
at gegl-sampler-linear.c:208
#1 0x00007ffff7b6a063 in gegl_sampler_get (self=self@entry=0x6c20a0 [GeglSamplerLinear], x=x@entry=2147483647, y=y@entry=2147483647, scale=scale@entry=0x0, output=output@entry=0x7fffffffdf60, repeat_mode=repeat_mode@entry=GEGL_ABYSS_NONE)
at gegl-sampler.c:178
#2 0x0000000000400bd7 in main (sampler_type=GEGL_SAMPLER_LINEAR)
at test-sampler.c:64
#3 0x0000000000400bd7 in main (argc=<optimized out>, argv=<optimized out>)
at test-sampler.c:94
and for the CUBIC sampler:
(gdb) run
Starting program: /home/simon/src/unstable/gegl/tests/simple/.libs/test-sampler
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
need (2147483646, 2147483646; 5, 5), have (0, 0; 0, 0)
0.000000,0.000000,0.000000,0.000000
need (2147483646, 85707686; 5, 5), have (0, 0; 0, 0)
fetching (2147483644, 85707684; 11, 11)
0.000000,0.000000,0.000000,0.000000
need (2147483646, 2147483646; 5, 5), have (2147483644, 85707684; 11, 11)
Program received signal SIGSEGV, Segmentation fault.
gegl_sampler_cubic_get (self=0x6c2080 [GeglSamplerCubic],
absolute_x=<optimized out>, absolute_y=<optimized out>,
scale=<optimized out>, output=0x7fffffffdf60, repeat_mode=GEGL_ABYSS_NONE)
at gegl-sampler-cubic.c:266
266 newval[0] += factor * sampler_bptr[0];
(gdb) bt
#0 0x00007ffff7b6b190 in gegl_sampler_cubic_get (self=0x6c2080 [GeglSamplerCubic], absolute_x=<optimized out>, absolute_y=<optimized out>, scale=<optimized out>, output=0x7fffffffdf60, repeat_mode=GEGL_ABYSS_NONE)
at gegl-sampler-cubic.c:266
#1 0x00007ffff7b6a063 in gegl_sampler_get (self=self@entry=0x6c2080 [GeglSamplerCubic], x=x@entry=2147483647, y=y@entry=2147483647, scale=scale@entry=0x0, output=output@entry=0x7fffffffdf60, repeat_mode=repeat_mode@entry=GEGL_ABYSS_NONE)
at gegl-sampler.c:178
#2 0x0000000000400bd7 in main (sampler_type=GEGL_SAMPLER_CUBIC)
at test-sampler.c:64
#3 0x0000000000400bd7 in main (argc=<optimized out>, argv=<optimized out>)
at test-sampler.c:94
Note that for the 1st coordinate it does not recognize, that it doesn't have the necessary area available. It does recognize this for the 2nd coordinate (where y is in a managable range), but on the 3rd attept it again doesn't recognize the need to fetch data. This time however, it crashes, since it probably tries to read way outside of the area cached.
Attachment 336440, "Demo program provoking the crash on 64 bit machines.":
test-sampler.c
Version: git master