Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Register
  • Sign in
  • G GLib
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 851
    • Issues 851
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 49
    • Merge requests 49
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GNOMEGNOME
  • GLib
  • Issues
  • #1775
Closed
Open
Issue created May 09, 2019 by Ghost User@ghostContributor

g_array_append_vals is very slow after 2Gb and corrupts data after 4Gb

If there is no enough allocated space in Garray, g_array_append_vals performs realloc. For array sizes under 2Gb size is doubled on each realloc. After 2Gb realloc performed for each elements (because g_nearest_pow returns 0) and it is very slow.

Another, but related bug - after array grows beyond 4Gb g_array_append_vals start to override existing values instead adding new.

Both issues can be demonstrated by attached test case: glib_array_append.c. Slowdown after 2Gb is especially clear if test case is compiled with LLVM address sanitizer. In fact g_array_append_vals became so slow after 2Gb, that it is almost impossible to use ASAN.

Changing type for memory size from guint to gsize will fix 2nd problem, but on 32bit platform append still will be slow after 2Gb.

Also doubling allocation size works well for small arrays but wastes a lot of memory for bigger ones. E. g. to store 1025 Mb array 2 Gb will be allocated and almost half of it will be wasted. A little bit better strategy is to double size for small sizes and increase allocation size by 25% for bigger ones (e. g. after some threshold from 256MB to 1Gb).

Even better allocation strategy will use malloc specific functions like nallocx in jemalloc to choose optimal allocation size, but as I can see glib doesn't do platform specific optimizations.

Edited May 09, 2019 by Ghost User
Assignee
Assign to
Time tracking