GIMP uses all memory on start and then crashes
Update 2: updated stacktraces
Update 3: json-c seems to be the bad guy (removed, before anyone puts brain power into this)
Update 4: json-c - still the bad guys! :)
Update 5: Over & Out
GIMP version: 2.9.8-r1 / 2.10.18-r1
Operating System: Gentoo/Linux 17.1/desktop (stable)
Package: as in original Gentoo portage tree
Description of the bug
Starting GIMP results in it using all memory: $ ps aux | gr gimp me 1420725 11.7 85.6 89048604 56382076 pts/8 Tl 13:57 1:20 gimp
Reproduction
Is the bug reproducible? Always
Reproduction steps:
- start gimp
If an ulimit is set (e.g. short of 90GB):
$ ulimit -v 89048470
$ gimp
will start as expected and consume virt/res 5449M / 160M.
I binary-searched this limit, but it fluctuates a little around this mark. Meaning:
89048470 always work
in between: good luck!
89048500 always does not
Increasing the ulimit a bit (or e.g. going straight to 100GB) causes the gimp process (and all of its workers) to consume up to virt/res 89GB / 58GB while res moves from ~3GB to 58GB, back to 3GB, rising to 58GB again, and so on.
I left it for ~15min to see whether it would start eventually, but it did not within this time.
Memory configuration of machine: 64GB RAM / 70GB SWAP
Disabling SWAP (obviously) circumvents the problem, as problem seems to occur at an available address space of ~90GB.
Running GIMP (without ulimit) under GDB and CTRL-C-ing at a random point always shows this backtrace:
(old stacktrace)
Update 2: recompiled 2.10.18-r1 (which is the current unstable on Gentoo), libmypaint and json-c with "-march=native -pipe -ggdb":
Thread 1 "gimp" received signal SIGINT, Interrupt.
0x00007ffff61e7478 in lh_table_new (size=2147483647, free_fn=0x0, hash_fn=0x7ffff61e72fa <lh_char_hash>, equal_fn=0x7ffff61e7362 <lh_char_equal>) at /var/tmp/portage/dev-libs/json-c-0.14-r2/work/json-c-0.14/linkhash.c:521
521 /var/tmp/portage/dev-libs/json-c-0.14-r2/work/json-c-0.14/linkhash.c: No such file or directory.
(gdb) bt
#0 0x00007ffff61e7478 in lh_table_new (size=2147483647, free_fn=0x0, hash_fn=0x7ffff61e72fa <lh_char_hash>, equal_fn=0x7ffff61e7362 <lh_char_equal>) at /var/tmp/portage/dev-libs/json-c-0.14-r2/work/json-c-0.14/linkhash.c:521
#1 0x00007ffff61e752f in lh_table_resize (t=0x555557ab1d50, new_size=2147483647) at /var/tmp/portage/dev-libs/json-c-0.14-r2/work/json-c-0.14/linkhash.c:540
#2 0x00007ffff61e771e in lh_table_insert_w_hash (t=0x555557ab1d50, k=0x555557b1b900, v=0x555557b1b5d0, h=4236111346, opts=0) at /var/tmp/portage/dev-libs/json-c-0.14-r2/work/json-c-0.14/linkhash.c:586
#3 0x00007ffff61dd9c3 in json_object_object_add_ex (jso=0x555557ab1cf0, key=0x555557ab2020 "custom_input_slowness", val=0x555557b1b5d0, opts=0) at /var/tmp/portage/dev-libs/json-c-0.14-r2/work/json-c-0.14/json_object.c:505
#4 0x00007ffff61dda24 in json_object_object_add (jso=0x555557ab1cf0, key=0x555557ab2020 "custom_input_slowness", val=0x555557b1b5d0) at /var/tmp/portage/dev-libs/json-c-0.14-r2/work/json-c-0.14/json_object.c:516
#5 0x00007ffff61e5767 in json_tokener_parse_ex
(tok=0x555557ab1400, str=0x555557aa89af ", \n \"dabs_per_actual_radius\": {\n", ' ' <repeats 12 times>, "\"base_value\": 2.0, \n", ' ' <repeats 12 times>, "\"inputs\": {}\n }, \n \"dabs_per_basic_radius\": {\n", ' ' <repeats 12 times>, "\"base_value\": 6.0, \n", ' ' <repeats 12 times>, "\"inputs\": {}\n"..., len=-1) at /var/tmp/portage/dev-libs/json-c-0.14-r2/work/json-c-0.14/json_tokener.c:1097
#6 0x00007ffff61e11ff in json_tokener_parse_verbose
(str=0x555557aa83b0 "{\n \"comment\": \"MyPaint brush file\", \n \"group\": \"\", \n \"parent_brush_name\": \"airbruch_press\", \n \"settings\": {\n \"anti_aliasing\": {\n", ' ' <repeats 12 times>, "\"base_value\": 1.39, \n", ' ' <repeats 12 times>, "\"inputs\""..., error=0x7fffffffc9dc) at /var/tmp/portage/dev-libs/json-c-0.14-r2/work/json-c-0.14/json_tokener.c:190
#7 0x00007ffff61e11a2 in json_tokener_parse
(str=0x555557aa83b0 "{\n \"comment\": \"MyPaint brush file\", \n \"group\": \"\", \n \"parent_brush_name\": \"airbruch_press\", \n \"settings\": {\n \"anti_aliasing\": {\n", ' ' <repeats 12 times>, "\"base_value\": 1.39, \n", ' ' <repeats 12 times>, "\"inputs\""...) at /var/tmp/portage/dev-libs/json-c-0.14-r2/work/json-c-0.14/json_tokener.c:178
#8 0x00007ffff70d54b1 in mypaint_brush_from_string
(self=0x555557787060, string=0x555557aa83b0 "{\n \"comment\": \"MyPaint brush file\", \n \"group\": \"\", \n \"parent_brush_name\": \"airbruch_press\", \n \"settings\": {\n \"anti_aliasing\": {\n", ' ' <repeats 12 times>, "\"base_value\": 1.39, \n", ' ' <repeats 12 times>, "\"inputs\""...) at mypaint-brush.c:1355
#9 0x0000555555afc87a in gimp_mybrush_load (context=0x555556207260, file=0x5555578b86c0, input=0x555556668f70, error=0x7fffffffcb40) at gimpmybrush-load.c:90
#10 0x0000555555a67229 in gimp_data_loader_factory_load_data (factory=0x5555561c61f0, context=0x555556207260, cache=0x0, dir_writable=0, file=0x5555578b86c0, info=0x55555667af80, top_directory=0x5555578b8480) at gimpdataloaderfactory.c:453
#11 0x0000555555a66fbf in gimp_data_loader_factory_load_directory (factory=0x5555561c61f0, context=0x555556207260, cache=0x0, dir_writable=0, directory=0x5555578b8800, top_directory=0x5555578b8480) at gimpdataloaderfactory.c:385
#12 0x0000555555a66f8b in gimp_data_loader_factory_load_directory (factory=0x5555561c61f0, context=0x555556207260, cache=0x0, dir_writable=0, directory=0x5555578b8480, top_directory=0x5555578b8480) at gimpdataloaderfactory.c:378
#13 0x0000555555a66e73 in gimp_data_loader_factory_load (factory=0x5555561c61f0, context=0x555556207260, cache=0x0) at gimpdataloaderfactory.c:330
#14 0x0000555555a6674e in gimp_data_loader_factory_data_init (factory=0x5555561c61f0, context=0x555556207260) at gimpdataloaderfactory.c:143
#15 0x0000555555a64b4c in gimp_data_factory_data_init (factory=0x5555561c61f0, context=0x555556207260, no_data=0) at gimpdatafactory.c:488
#16 0x0000555555a104fc in gimp_data_factories_load (gimp=0x5555561bc020, status_callback=0x555555638f66 <splash_update>) at gimp-data-factories.c:360
#17 0x0000555555a0c2da in gimp_restore (gimp=0x5555561bc020, status_callback=0x555555638f66 <splash_update>, error=0x7fffffffce80) at gimp.c:806
#18 0x000055555562eec4 in app_run
(full_prog_name=0x555556014480 "/usr/bin/gimp", filenames=0x0, alternate_system_gimprc=0x0, alternate_gimprc=0x0, session_name=0x0, batch_interpreter=0x0, batch_commands=0x0, as_new=0, no_interface=0, no_data=0, no_fonts=0, no_splash=0, be_verbose=0, use_shm=1, use_cpu_accel=1, console_messages=0, use_debug_handler=0, show_playground=0, show_debug_menu=0, stack_trace_mode=GIMP_STACK_TRACE_NEVER, pdb_compat_mode=GIMP_PDB_COMPAT_ON, backtrace_file=0x555556013690 "/home/nomahlo/.config/GIMP/2.10/CrashLog/GIMP-crash-1589808712.txt") at app.c:338
#19 0x0000555555632fdf in main (argc=1, argv=0x555556014270) at main.c:636
Here is a backtrace of one of its worker threads at that time:
Update 2: for the sake of it, I also recompiled glibc und gegl to debug
(gdb) bt
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007ffff6faeccf in g_cond_wait () at /usr/lib64/libglib-2.0.so.0
#2 0x00007ffff745787c in gegl_parallel_distribute_thread_func (thread=0x7ffff751c1d8 <gegl_parallel_distribute_threads+56>) at ../gegl-0.4.22/gegl/gegl-parallel.c:508
#3 0x00007ffff6f8b36d in g_thread_proxy () at /usr/lib64/libglib-2.0.so.0
#4 0x00007ffff6db7010 in start_thread (arg=<optimized out>) at pthread_create.c:479
#5 0x00007ffff6cedb0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
$ ps huxaH | cat | awk '{print $1 "\t" $11}' | sort -k 2 | uniq -c | sort -h | cat -n
lists 68 threads for gimp at this point
(I removed all .gimp... / .config/GIMP directories to make sure it's not a 'state')
I also recompiled/re-emerged libjson-c, libmypaint and gimp to rule out a linkage problem, but without success.
Update 3 (removed, as this seems to be a compile/link issue on my side)
Update 4
Ok, while hanging on this extreme mem-load, breaking leads to this piece here:
(from json-c-0.14/linkhash.c
)
578 int lh_table_insert_w_hash(struct lh_table *t, const void *k, const void *v, const unsigned long h,
579 const unsigned opts)
580 {
581 unsigned long n;
582
583 if (t->count >= t->size * LH_LOAD_FACTOR) {
584 /* Avoid signed integer overflow with large tables. */
585 int new_size = INT_MAX / 2 < t->size ? t->size * 2 : INT_MAX;
586 if (t->size == INT_MAX || lh_table_resize(t, new_size) != 0)
587 return -1;
588 }
589
590 n = h % t->size;
591
Line 585 struck me: LH_LOAD_FACTOR
is 0.66
, t->count
is 11
and t->size
is 16
.
Now int(16 * 0.66) == 10
, so the next test checks, whether table's size is already bigger than INT_MAX / 2
.
If it is not, it can be doubled, otherwise INT_MAX
would be the next possible size, but this test does it the other way around!
So I changed the <
into an >
, recompiled and Gimp starts just fine.
I have no idea why this only creates a problem for >89 GB RAM..
Yet, if this is really the offending piece of code, I need to file it as a json-c bug.
Update 5
Ok, adding this comparison-invert-patch and recompiling everything back to release works - no matter the setting.
I still have no idea why this is v-address-space related, but... meh.
Thanks to @aklapper and @Jehan for helping me into the right direction!
-> json-c bug. Sorry for the fuss! Closing..