g_array_append_vals is very slow after 2Gb and corrupts data after 4Gb
If there is no enough allocated space in Garray
, g_array_append_vals
performs realloc. For array sizes under 2Gb size is doubled on each realloc. After 2Gb realloc performed for each elements (because g_nearest_pow returns 0) and it is very slow.
Another, but related bug - after array grows beyond 4Gb g_array_append_vals
start to override existing values instead adding new.
Both issues can be demonstrated by attached test case: glib_array_append.c.
Slowdown after 2Gb is especially clear if test case is compiled with LLVM address sanitizer. In fact g_array_append_vals
became so slow after 2Gb, that it is almost impossible to use ASAN.
Changing type for memory size from guint
to gsize
will fix 2nd problem, but on 32bit platform append still will be slow after 2Gb.
Also doubling allocation size works well for small arrays but wastes a lot of memory for bigger ones. E. g. to store 1025 Mb array 2 Gb will be allocated and almost half of it will be wasted. A little bit better strategy is to double size for small sizes and increase allocation size by 25% for bigger ones (e. g. after some threshold from 256MB to 1Gb).
Even better allocation strategy will use malloc specific functions like nallocx in jemalloc to choose optimal allocation size, but as I can see glib doesn't do platform specific optimizations.