large import uses lots of memory, sometimes leading to crash
Submitted by Adam Dingle
Assigned to Jim Nelson
Link to original bug (#716840)
Description
---- Reported by adam@yorba.org 2010-09-16 08:54:00 -0700 ----
Original Redmine bug id: 2566
Original URL: http://redmine.yorba.org/issues/2566
Searchable id: yorba-bug-2566
Original author: Adam Dingle
Original description:
From Joao Coelho norphon@hotmail.com:
On Ubuntu 10.04 tried to import 11,000 photos from F-Spot database. First stage completed successfully. Crashed during second stage where photos are flashed in sequence and events and tags are created in Shotwell. Memory use keeps climbing until system runs out of memory and it all falls over.
---- Additional Comments From shotwell-maint@gnome.bugs 2013-05-01 11:40:00 -0700 ----
History
Comment 1
Updated by Jim Nelson about 3 years ago
A user on Launchpad reported something similar: https://bugs.launchpad.net/bugs/588569
Comment 2
Updated by Vera Yin about 3 years ago
I've done some tests on my system (Ubuntu 10.10 beta) and observed the following:
a) importing an F-Spot library of 10,000 photos in 0.7.2 – memory use climbs to 1GB halfway through and stays there
b) importing a folder of 10,000 photos in 0.7.2 – memory use climbs to 700MB halfway through and stays there
c) importing a folder of 10,000 photos in trunk – memory use climbs to 1GB 40% of the way through, drops to 800MB by the end
Comment 3
Updated by Jim Nelson about 3 years ago
Marked #2349 (closed) as a duplicate.
Comment 4
Updated by Jim Nelson about 3 years ago
- Status changed from Open to 5
- Resolution set to fixed
- % Done set to 100
Comment 5
Updated by Vera Yin about 3 years ago
Some updated numbers after Jim's fix -
Import of 10,000 photos (from disk, linking) took 18 minutes, memory usage peaking at 160MB.
Import of 10,000 photos (from disk, copying) took 31 minutes, memory usage peaking at 135MB.
Import of 157 photos (from camera) took 30 seconds, memory usage peaking at 80MB.
Comment 6
Updated by Jim Nelson about 3 years ago
Fantastic. This is well within normal bounds.
Comment 7
Updated by Mike - about 3 years ago
- Status changed from 5 to 4
-
Resolution deleted (
<strike>
_fixed_</strike>
) - % Done changed from 100 to 0
-
Priority deleted (
<strike>
_Urgent_</strike>
)
I had a rather large collection to import (40K+ images), and it took the repo Shotwells three imports to do them (the first two caused memory usage grow so large as to crash the program AND my window compositor (!)).
The bad news: I'm still noticing the issue in trunk. I downloaded, configured, and compiled this morning, then when I do a sizeable import, the window becomes unresponsive, and my memory usage climbed to nearly a gigabyte (989MiB). This was just importing around 1/20th of my library; I suspect an attempted full import would cause the same issue I was seeing before. Moreover once the import concluded, the memory used during import was not released.
How can I help you identify and fix this here? By the way, I was very reluctant to reopen this, but I want to make sure we get this bug squashed; this will be a hindrance to adoption of previous F-Spot users like myself.
Comment 8
Updated by Mike - about 3 years ago
I can provide some more info. I just reproduced my issue:
Import (from disk, linking) of 5,544 images took approximately 20 minutes and memory usage peaked (and remained) at 1.2GiB.
Shotwell 0.7.2+trunk
Ubuntu 10.04
Kernel 2.6.32.25
2.0 GiB memory
Athlon 64 processor 3200+
Comment 9
Updated by Jim Nelson about 3 years ago
- Priority set to Urgent
Hmm … definitely concerned about you seeing this running from trunk. One thing first off: What version of gexiv2 are you using?
$ pkg-config --modversion gexiv2
We had a memory leak there as well a while back. Would be best if you were using 0.2.1 (or better, trunk) to see if this problem persists.
Also, are you importing videos? If so, how many?
Comment 10
Updated by Mike - about 3 years ago
I'm using gexiv2 0.2.1. That was 5544 images and 114 videos.
Comment 11
Updated by Adam Dingle about 3 years ago
- Subject changed from large import from F-Spot crashes to large import uses lots of memory, sometimes leading to crash
Renaming ticket to indicate this this is not F-Spot specific and involves a memory leak.
Comment 12
Updated by Jim Nelson about 3 years ago
- Status changed from 4 to 5
- Resolution set to fixed
- % Done changed from 0 to 100
r2371
Comment 13
Updated by Adam Dingle about 3 years ago
- Status changed from 5 to 4
-
Resolution deleted (
<strike>
_fixed_</strike>
) - % Done changed from 100 to 0
I had to back outr2371and the associatedr2374because they caused a hang on import; see#2864. So I'm reopening this ticket. Jim can look at this when he's back at Yorba later this week.
Comment 14
Updated by Jim Nelson almost 3 years ago
- Status changed from 4 to 5
- Resolution set to fixed
- % Done changed from 0 to 100
Threads could become starved in certain cases (race condition). Also discovered a problem with a non-thread-safe code being called by background threads; this only occurs when files are copied into the library during import.
This patch should solve the starvation and locking problems. r2387
Comment 15
Updated by Charles Lindsay 7 months ago
- Status changed from 5 to Fixed
--- Bug imported by chaz@yorba.org 2013-11-25 21:47 UTC ---
This bug was previously known as bug 2566 at http://redmine.yorba.org/show_bug.cgi?id=2566
Unknown Component Using default product and component set in Parameters Unknown version " in product shotwell. Setting version to "!unspecified". Unknown milestone "unknown in product shotwell. Setting to default milestone for this product, "---". Setting qa contact to the default for this product. This bug either had no qa contact or an invalid one.
Resolution: RESOLVED FIXED