RAW thumbnail generation is CPU intensive
Submitted by an unknown user
Link to original bug (#717532)
Description
---- Reported by shotwell-maint@gnome.bugs 2011-05-07 22:25:00 -0700 ----
Original Redmine bug id: 3586
Original URL: http://redmine.yorba.org/issues/3586
Searchable id: yorba-bug-3586
Original author: Pedro C ôrte-Real
Original description:
I decided to test shotwell by importing all my library into it. It consists of 300GB in 38k files of which 19k are RAW files. I currently manage it with f-spot and imported in place (without copying). The test was done on a Lenovo X61 with a 2.2GHz Core2Duo and shotwell 0.9.2 as shipped by ubuntu natty. Shotwell took around 15 hours to do the full import and that's without generating all the mimics. This seemed excessive so I did some tests on the same machine:
Using the dcraw settings f-spot uses and piping that into convert:
$time for i in find ~/Photos/ -name "*.ARW" | sort -R | tail -n 100
; do
dcraw -h -w -c -t 0 $i | convert -geometry 128×128 - $(basename $i).jpg ;
done
real 1m44.282s
user 1m4.250s
sys 0m10.490s
So for 38k photos it would be 1.75m*380 = 11 hours, which sounds around the right ballpark for what we got.
Then I tested importing 2GB of RAW photos into both f-spot and shotwell. The results were:
Importing from CF card (so needing to copy)
f-spot: ~3m10s with <50% of a core used
shotwell: ~3m40s with 100% of a core used
Importing from ~/Pictures (in place)
f-spot: ~1m20s with 100% of a core used
shotwell: ~4m07s with <100% of a core used (that this ended up slower is odd and probably just means my methodology was too sloppy)
Clearly there's some improvement potential here. f-spot is doing the half-size RAW conversion trick (-h) but so is shotwell I think. I wonder if there's any technical limitation to doing raw conversions in quarter or eigth size to speed up conversions more. It could also help with not needing mimics any more.
Related issues:
- related to shotwell - 3051: Camera timing out during import (Open)
- related to shotwell - 6118: Excessive memory use on changing developer (Open)
- related to shotwell - 1856: System sluggish during a drag-and-drop RAW import (Open)
---- Additional Comments From shotwell-maint@gnome.bugs 2013-05-16 14:44:00 -0700 ----
History
Comment 1
Updated by Pedro Côrte-Real over 2 years ago
Note that one reason f-spot probably saturates the CPU a bit better is that they fork out for dcraw so they probably end up using the two cores, one for the raw conversions, another for everything else. They don't seem to be forking more than one conversion at a time though. This isn't enough to explain the difference though as there's more than a 2x difference between f-spot and shotwell.
Comment 2
Updated by Adam Dingle over 2 years ago
- Priority set to High
Would be good to investigate this more.
Comment 3
Updated by Lucas Beeler about 2 years ago
- Description updated (diff)
- Target version set to 0.12
Since we're going to be digging into the RAW developer stuff again in 0.12, we might be able to attack this.
Comment 4
Updated by Jim Nelson about 2 years ago
- Description updated (diff)
-
Target version deleted (
<strike>
_0.12_</strike>
)
Comment 5
Updated by Jim Nelson 11 months ago
- Target version set to 0.14.0
Comment 6
Updated by Jim Nelson 11 months ago
- Category set to raw
Comment 7
Updated by Jim Nelson 11 months ago
- Tracker changed from Feature to Bug
- Priority changed from High to Normal
We should retest this with latest Shotwell and LibRaw and see how it looks.
Comment 8
Updated by Jim Nelson 10 months ago
- Target version changed from 0.14.0 to 0.15.0
Comment 9
Updated by Jim Nelson 10 months ago
Also, this may be a duplicate of #1856 (closed).
Comment 10
Updated by Jim Nelson 8 months ago
- Target version changed from 0.15.0 to 0.16.0
Comment 11
Updated by Jim Nelson 6 months ago
-
Target version deleted (
<strike>
_0.16.0_</strike>
)
--- Bug imported by chaz@yorba.org 2013-11-25 21:52 UTC ---
This bug was previously known as bug 3586 at http://redmine.yorba.org/show_bug.cgi?id=3586
Unknown version " in product shotwell. Setting version to "!unspecified". Unknown milestone "unknown in product shotwell. Setting to default milestone for this product, "---". Setting qa contact to the default for this product. This bug either had no qa contact or an invalid one. Resolution set on an open status. Dropping resolution