Photos take 10 minutes to appear on start-up
Submitted by an unknown user
Assigned to Jim Nelson
Link to original bug (#716886)
Description
---- Reported by shotwell-maint@gnome.bugs 2010-09-03 08:26:00 -0700 ----
Original Redmine bug id: 2515
Original URL: http://redmine.yorba.org/issues/2515
Searchable id: yorba-bug-2515
Original author: Adam Hooper
Original description:
Steps to reproduce:
-
Get a reasonably large photo library (mine's 7,000 strong)
-
Close Shotwell
-
Open Shotwell
Expected results: all photos and thumbnails appear within seconds and Shotwell is interactive.
Actual results: the photos appear slowly, 50 at a time (about 50 every 2 seconds on my netbook), with no progress bar and no indication that more are about to appear. Shotwell is slow to respond to my clicks. On my computer, it takes about 10 minutes before I can browse my photo library.
Why: I'm guessing Shotwell inspects each file in the database before displaying it.
(If my guess is correct, why is there a database at all if Shotwell is just going to load photo info from the original files every time it starts? If Shotwell already has a list of all the photos and their metadata and has already generated their thumbnails, I expect basic browsing to be instantaneous.)
This bug makes me unable to, say, quickly start Shotwell and show my friends a slide-show I've tagged ahead of time. My workaround is to export my photos to a separate directory as soon as I finish tagging them, so I can use Eye of Gnome when my friends are around.
---- Additional Comments From shotwell-maint@gnome.bugs 2013-05-01 11:47:00 -0700 ----
History
Comment 1
Updated by Adam Dingle about 3 years ago
Adam,
what version of Shotwell are you running, and which Linux distribution? What make/model of netbook are you using?
Comment 2
Updated by Adam Hooper about 3 years ago
- Subject changed from Photos take forever to appear on start-up to Photos take 10 minutes to appear on start-up
Shotwell 0.7.1 as distributed on Ubuntu 10.10 alpha with a Gateway LT31u (1.2Ghz Athlon L110 processor, 2GB RAM).
Comment 3
Updated by Adam Dingle about 3 years ago
Thanks for the info. The behavior you describe is unusual: on most machines Shotwell displays all photos and is responsive within a few seconds, even with a large photo library. Shotwell does not load photo info from the original files each time it starts – it should read only the thumbnail files.
During the 10 minutes before Shotwell is responsive, is Shotwell using CPU time? To see this, start System->Administration->System Monitor and choose the Process tab, then look at the shotwell process. Is its % CPU at 100%?
Comment 4
Updated by Adam Hooper about 3 years ago
Yup, it never wavers below around 98%.
My photo library is almost exclusively NEFs, if that makes a difference.
I'll install sysprof and see if it gives a hint as to what's taking so long.
Comment 5
Updated by Adam Dingle about 3 years ago
sysprof data would be very helpful here. You might need to build Shotwell from source in order to get meaningful output, since I'm not sure whether Ubuntu includes debug symbols in the binary it distributes. Our Web site explains how to do that; if you need any help just let us know.
sysprof lets you save the profiling output to a file; it would be great if you could attach that here. I don't think you need to profile for all 10 minutes that Shotwell is unresponsive; just a 10- or 20-second interval should be revealing enough. Thanks!
Comment 6
Updated by Adam Hooper about 3 years ago
- Status changed from Open to 5
- Resolution set to invalid
- % Done set to 0
Oh hey, what do you know: when built from source this bug doesn't exist. I'll move this bug downstream--thanks!
Comment 7
Updated by Adam Dingle about 3 years ago
Hm… this still seems odd. I know that Ubuntu has applied some small patches to its own build of Shotwell, but I can't imagine they would cause this behavior.
I wonder if Shotwell was performing some background processing on the NEF files which made it slow, and whether it's faster now because that processing is done (not because you happen to be running a different binary). Can you double-check and confirm that with your current Shotwell library, if you run the Ubuntu binary it takes 10 minutes before Shotwell is responsive, but if you run the home-built binary then it responds right away?
Comment 8
Updated by Adam Dingle about 3 years ago
- Status changed from 5 to 4
-
Resolution deleted (
<strike>
_invalid_</strike>
) - % Done changed from 0 to 0
Reopening because even if this occurs only with the Ubuntu binary we still want to help out in tracking this down.
Comment 9
Updated by Adam Hooper about 3 years ago
Still happens. I'll revert my svn checkout to 0.7.1 and see if the bug was in that release. Any other suggestions?
Comment 10
Updated by Adam Hooper about 3 years ago
I've attached a sysprof profile, for what it's worth, taken while around 1000-2000 photos were appearing.
Comment 11
Updated by Adam Dingle about 3 years ago
Oh – you built from trunk! :) There have been major changes in trunk since 0.7.1, so I think you should definitely build the 0.7.1 tarball and see what happens there.
Comment 12
Updated by Adam Dingle about 3 years ago
I looked at the sysprof profile you attached, but unfortunately it's not very revealing because no Shotwell debug symbols were present. If you do see the slowness with your own 0.7.1 build, it would be very interesting to see another sysprof profile generated with that binary.
Comment 13
Updated by Adam Hooper about 3 years ago
It does happen on 0.7.1 when built from source. I've attached a sysprof profile (again, not sure how useful it is).
To me, what jumps out is “thumbnail_on_tag_contents_altered†… but I don't know the Shotwell architecture so I don't know if that call is intentional.
Comment 14
Updated by Adam Dingle about 3 years ago
Thanks for this latest profile. Unfortunately it still doesn't contain a very descriptive call stack, probably because debug symbols are missing for the various system libraries. Could you possibly install the following packages, then run sysprof again and attach the output?
- libc6-dbg
- libgtk2.0-0-dbg
- libglib2.0-0-dbg
Comment 15
Updated by Adam Hooper about 3 years ago
I've installed all the debugging libraries I could find in Ubuntu and it still doesn't expand into a tree like it should. Anyway, I've updated the sysprof profile.
I've also run in gdb to get a typical backtrace. (I paused it a couple of times, it looked the same.)
One other symptom: I'm not sure I saw this every time, but sometimes when the “Basic Information†pane first appears it'll have the proper photo count for a fraction of a second, then it'll go blank and start counting: 0, 50, 100, ….
Comment 16
Updated by Adam Dingle about 3 years ago
-
Priority deleted (
<strike>
__</strike>
)
Hm – I don't understand why the sysprof profile isn't expanding into a tree even with all the debug symbols present. Oh, well – the gdb backtrace you sent should be good enough. We'll look at that and see if we can figure out what's going on.
Upping priority to release since this seems important to fix.
Comment 17
Updated by Adam Hooper about 3 years ago
Hrm. Dunno how else I can help, but ask if you think of anything.
Comment 18
Updated by Jim Nelson about 3 years ago
Adam Hooper,
Where are your files located? That is, are they on an external drive or a network drive?
Shotwell does scan your photos at startup, but it's not examining each one intensely (that's what the import is for). At startup, 0.7.1. merely scans the photos to verify their presence; it marks missing ones as “offline†and ones that were previously missing (but now detected) as “onlineâ€. This was in response to users who complained that they were deleting or moving photos directly in the file system and not seeing the changes reflected in Shotwell.
When photos are marked offline, you'll see a “Missing Files†page near the bottom of the sidebar, next to the Trash.
Looking at your stack trace, it appears many (all?) of your files have been marked as offline and are now being discovered and reset to online. The reason you're seeing them come online in batches of 50 is that it's too slow to mark each one one at a time, but to wait to mark all 7,000 at once causes the UI to hang. So they're done in clusters.
If this is happening almost every time you run Shotwell, it indicates to me that there's some reason Shotwell doesn't detect the photos en masse and is marking them offline, which takes time. Then, the next time you run it, it does see them and goes through the process again marking them online. This is why I ask where they're located; if they're on a flaky network connection or an external drive that's coming and going, this is exactly what I would expect.
I'm not happy it's taking 10 minutes, mind you. But looking at the stack trace and reading your comments, this is what it looks like the problem is.
Also, Shotwell generates full-sized images called “mimics†for RAW files which it uses for quick viewing and interactive editing of your photos. These mimics are generated in the background. How long has it been since you imported your photos? I'm concerned that the mimics are still being generated while these photos are being marked online and that's causing your CPU to spike as well. One thing you can do is this:
$ find ~/.shotwell/mimics | wc
The first number should approximate the number of NEF files you have in your library. If it's seriously less, then that means Shotwell is still generating the mimics.
Thanks for your quick response and all the information you've been providing.
Comment 19
Updated by Adam Hooper about 3 years ago
Hi jim,
The files are all on my hard drive, but they're on another partition (my ~/Pictures is a symlink).
I've made an altered database which points to nonexistent files, and when opening it I've watched the number of files drop (7,000 – 6,950 – 6,900 – 6,850 – etc.). The number dropped very quickly--maybe 10 times faster than when it rises. (I guess that's what you'd expect.)
I have 2444 NEFs and 2426 mimics.
Comment 20
Updated by Adam Dingle about 3 years ago
Jim can now reproduce this here! It's the symbolic link that's causing the trouble.
Comment 21
Updated by Jim Nelson about 3 years ago
- Status changed from 4 to Review
- Assignee changed from Anonymous to Jim Nelson
I've committed to the 0.7 branch a fix for this problem. It's due to how were were not (fully) supporting symbolic links during the startup scan.
Adam, are you working out of our Subversion repository? If you pull Shotwell from the branch and build it, I would be interested to know if your problem is solved:
$ svn co svn://svn.yorba.org/shotwell/branches/shotwell-0.7
Thanks,
-- Jim
Comment 22
Updated by Adam Hooper about 3 years ago
It works! Thanks. And as I mentioned above, it already works on trunk as well.
Comment 23
Updated by Jim Nelson about 3 years ago
- Status changed from Review to 5
- Resolution set to fixed
- % Done changed from 0 to 100
Great! Thanks for all your help.
Comment 24
Updated by Vera Yin about 3 years ago
- Resolution changed from fixed to verified
-
% Done deleted (
<strike>
_100_</strike>
)
Comment 25
Updated by Vera Yin about 3 years ago
- Resolution changed from verified to fixverified
-
% Done deleted (
<strike>
__</strike>
)
Comment 26
Updated by Charles Lindsay 7 months ago
- Status changed from 5 to Fixed
--- Bug imported by chaz@yorba.org 2013-11-25 21:47 UTC ---
This bug was previously known as bug 2515 at http://redmine.yorba.org/show_bug.cgi?id=2515 Imported an attachment (id=261844) Imported an attachment (id=261845) Imported an attachment (id=261846)
Unknown Component Using default product and component set in Parameters Unknown version " in product shotwell. Setting version to "!unspecified". Unknown milestone "unknown in product shotwell. Setting to default milestone for this product, "---". Setting qa contact to the default for this product. This bug either had no qa contact or an invalid one.
Resolution: RESOLVED FIXED