Initial load of folders with many items is too slow on HDDs
This is complementary (but apparently different) to issues #3062 and #3067. Here, search is not involved, and scrolling is not involved. It is purely about processing performance for local folders with many items.
Summary: Nautilus is much slower to show (and to an extent load) folders with hundreds/thousands of items, compared to XFCE's "Thunar" file manager. It seems to be in big part due to thumbnails attributes processing being expensive.
- Particularly on HDDs, and particularly on cold start (i.e. not visiting a folder you previously visited during the session)
- Even with thumbnails disabled (and thumbnails cache already fully generated)
- Even with subfolder counts disabled (or no subfolders present at all)
- Even with
noatime
in/etc/fstab
Click to expand: hardware and methodology used for these benchmarks
## Testing parameters
My two benchmark machines, both with roughly the same "Downloads" folder containing about 2000 items:
- My current desktop workstation, with a new/fast HDD
(I use that instead of a SSD to contain huge amounts of heavy data files), with Nautilus nightly flatpak, 24 GB of RAM, and a beefier CPU. Note that the nightly flatpak has broken thumbnails on that machine (and otherwise that machine's thumbnails has its thumbnails cache directory on a SSD, not the HDD) - My previous desktop computer, with an old HDD and 5 GB of RAM, using native (RPM) Nautilus 46 from Fedora 40.
It where I ran sysprof; I used this slower computer to exhibit performance problems more clearly.
RAM usage remained under 2 GB used out of 5 GB at all times.
Both computers use the ext4
filesystem with LUKS encryption. The fast-HDD computer has noatime
in /etc/fstab
, the older computer with the slower HDD does not have noatime (or anything similar) configured.
Methodology
- Measuring Nautilus 46 vs Thunar's time to 1) start showing contents of the folder 2) complete any remaining processing
- I measured multiple times, with a stopwatch, and sysprof 46
- To ensure consistently reproducible measurements, I flushed the kernel's disk caches to ensure we're truly reading "cold" from the disk.
- I also measured "warm" times, but those are less relevant.
- GIO measurements using this Python script (but results are worse than real-world scenarios)
To flush disk caches each time, I use this trick (to be run as root):
sync && sync && echo 3 > /proc/sys/vm/drop_caches
Summary comparison of load times (in seconds)
Listview, thumbnails off, folders counts off | Nautilus 46 | Thunar w/ thumbs | Thunar no thumbs | GIO Py script, with thumbs | GIO Py script, no thumbs | |
---|---|---|---|---|---|---|
Cold cache – old HDD, 1850 files, RPM | Time for initial view/items to appear | 32 | 0.9 | 0.7 | NA | NA |
Time for folder to be fully processed | 33 | 8.5 | 4.3 | 24.59 | 0.00071 | |
Warm cache - old HDD, 1850 files, RPM | Time for initial view/items to appear | 1.7 | 0.5 | 0.5 | NA | NA |
Time for folder to be fully processed | 2.0 | 1.6 | 1.6 | 0.37 | 0.00070 | |
Cold cache – new HDD, 2200 files, Flatpak | Time for initial view/items to appear | 7.5 | 5.4 | 3.1 | not tested | not tested |
Time for folder to be fully processed | 8.2 | 6.7 | 4.4 | not tested | not tested | |
Warm cache – new HDD, 2200 files, Flatpak | Time for initial view/items to appear | 1.7 | 0.4 | 0.4 | not tested | not tested |
Time for folder to be fully processed | 2.0 | 1.8 | 1.8 | not tested | not tested |
Sysprof captures from the old HDD
These captures are with the thumbnails cache already fully generated beforehand, and thumbnails turned off entirely, to eliminate those from the equation.
Sysprof capture file: Nautilus_46_listview_loading_a_2000_items_folder__after_completed_thumbnailing.tar.xz
Flame graph of the first 90% part (where Nautilus shows nothing while processing the folder's files), excluding the actual view's graphical loading:
Click to expand: Complete flame graph (including the view's painting after files have been processed)
Hard drive disk IO usage chart over time:
iotop idle before loading the folder | iotop during loading, snapshot 1 | iotop during loading, snapshot 2 | iotop after folder is loaded |
---|---|---|---|
Access times (for the thumbnail_verify
function that compares the cached thumbnail files mtime with the source file's mtime) seem to mostly follow a normal distribution:
…with times usually between 10 and 30 miliseconds (and sometimes between 30 and 70 ms), when you do that thousands of times, it adds up (note: the graph above is sorted by times; not chronological and not cumulative).
Observations
- With a cold disk kernel cache, Thunar consistently loads any folder faster than Nautilus: between 4x and 45x faster, depending on how you look at it
- Thunar always starts displaying folder contents within 1.2 seconds (even if it's just the currently visible part of the view), and keeps processing the rest (if any) in the background. During that time, Nautilus does not even show a progressbar (which could at least help users wait more patiently, compared to a spinner). There was !1162 (closed) that proposed refreshing the view incrementally, but it was closed as it was unclear then whether it would be reliable and applicable after 2024 Q4's refactoring efforts.
- Based on what I heard from the hard drive's head during my tests, Thunar seems to grind the disk much less, while Nautilus seems to grind much more intensively and longer.
- According to the flamegraph, Nautilus spends most of its time processing thumbnail info, even when it doesn't strictly need to (if thumbnails are disabled). One could think we could just avoid touching thumbnails code in that case. However:
- even if Nautilus' "zoomed out listview" displays icons instead of thumbnails (via CSS), it's only a zoom-in away from needing to instantly display them… so that might not be a silver bullet;
- browsing "with thumbnails fully disabled" otherwise is kind of a niche usecase;
- Thunar manages to do all this with thumbnails activated (even if it has to generate them), whereas Nautilus struggles even with thumbnails deactivated and hidden. Ideally we should figure out a way for Nautilus'
get_thumbnail_attributes
/thumbnail_verify
to be fast instead, as it would presumably speed up Nautilus' views everywhere (including search).
Summary: overall, in terms of UX, a folder that Nautilus will take many seconds (or even minutes) before loading or starting to show contents, will feel "instantaneous" to load in Thunar, in part because it starts showing contents early, and in part because it seems to be more efficient, so presumably Nautilus is also doing extra work that it shouldn't be doing.