import uses an enormous amount of memory, leading to crash
Submitted by Adam Dingle
Link to original bug (#718134)
Description
---- Reported by adam@yorba.org 2011-12-16 09:35:00 -0800 ----
Original Redmine bug id: 4498
Original URL: http://redmine.yorba.org/issues/4498
Searchable id: yorba-bug-4498
Original author: Adam Dingle
Original description:
Josep Cols reported this on the mailing list (see http://lists.yorba.org/pipermail/shotwell/2011-December/003455.html):
It's a new shotwell 0.11.6-1-natty1 install on a Ubuntu 11.04.
I'm importing 25.000 photos from my f-spot database.
The first try, about 5.100 photos was imported before shotwell crash.
After crash, a pop-up window says something about 'no memory available'
(sorry, the original message was in catalan...)
===
I've tried to import 5 times... the last 2 imports, shotwell used about 3.1 Gb or Virtual Memory before crash. I think that 3 Gb is the limit for the virtual memory that can use a 32 bits program...
Related issues:
- related to shotwell - Feature #3614 (closed): Provide an API for external program import plugins (Fixed)
---- Additional Comments From shotwell-maint@gnome.bugs 2013-05-01 11:45:00 -0700 ----
History
Comment 1
Updated by Adam Dingle almost 2 years ago
More information from Josep:
The last 2 imports (when 3.1 Gb of virtual memory is used) imported only 1.200 photos every import.
I think the quantity of photos imported and/or the amount of memory used is due to the tags.
I've about 10.000 tags at the last level and about 13.000 tags in total on the database. The low level tags has 3-5 levels.
Every photo as a minimum of 6 tags and an average of 8-10 tags (and a maximum of 20 tags).
Also, the photos are jpg from 10 OR 15 Mpixels camera (between 4 and 8 Mbytes each).
If you need more information, I can run tests for you and/or install specific software for test purposes, etc.
Comment 2
Updated by Josep Cols Canals almost 2 years ago
Hello:
I can reproduce this memory problem simply adding photos to shotwell (it is not necessary to import them from f-spot).
Also, importing photos with no tags does not reproduce the problem.
Comment 3
Updated by Adam Dingle almost 2 years ago
In a followup email, Josep wrote:
To reproduce this problem, simply use a photo that has a IPTC tag not present as [XMP] one.
Comment 4
Updated by Adam Dingle almost 2 years ago
- Subject changed from F-Spot import uses an enormous amount of memory, leading to crash to import uses an enormous amount of memory, leading to crash
Comment 5
Updated by Josep Cols Canals almost 2 years ago
Just to clarify:
It was a mistake in emails.
I can reproduce this issue importing photos that has KEYWORDS defined (XMP and/or IPCT) into shotwell.
In my environmemt, I have a lot of TAGS (10.000) on my database and every photo has from 6 to 12 KEYWORDS defined. Now, I have 15.000 photos already imported into shotwell and shotwell crashes, due to the memory used, after 300 new photos are imported.
(I can't reproduce importing photos with differents IPTC and XMP KEYWORDS.
Comment 6
Updated by Bruno Girin almost 2 years ago
The main problem is that the F-Spot import feature loads the whole F-Spot database in memory and performs a single large batch import. So even if it was tuned, there would always be a possibility that a very large database would make it run out of memory. So the only way to fix this is to completely overhaul the code.
The good news is that this code overhaul is happening right now. I've made good progress during the Christmas break on 3614 (http://redmine.yorba.org/issues/3614) to move the code to the SPIT architecture. As part of that work, I've changed the behaviour to avoid importing everything in one go. There are still a few rough edges but I should have a first patch for review in the next couple of weeks. At that point, it would be very useful to test the functionality on large databases to see if it actually fixes the problem.
Comment 7
Updated by Josep Cols Canals almost 2 years ago
When you have a beta-patch, I can use my database and photo collection (26.000 photos) to test the patch.
Comment 8
Updated by Bruno Girin almost 2 years ago
Thanks Josep, I added a relationship with ticket 3614 as it has the potential to fix this one.
Comment 9
Updated by Adam Dingle over 1 year ago
-
Target version deleted (
<strike>
_0.12_</strike>
)
Comment 10
Updated by Jim Nelson 11 months ago
- Target version set to 0.14.0
Comment 11
Updated by Jim Nelson 11 months ago
- Category set to import
- Status changed from Open to Need Information
Josep, it's been a while since Bruno update Shotwell to the new F-Spot import system. Can you try and reproduce this problem?
Comment 12
Updated by Jim Nelson 8 months ago
- Target version changed from 0.14.0 to 0.15.0
Comment 13
Updated by Jim Nelson 8 months ago
- Status changed from Need Information to 5
- Resolution set to invalid
Closing as this ticket has been Need Information for almost 90 days now. Please re-open if this problem is discovered again or more information comes to light.
Comment 14
Updated by Charles Lindsay 7 months ago
- Status changed from 5 to Invalid
--- Bug imported by chaz@yorba.org 2013-11-25 21:55 UTC ---
This bug was previously known as bug 4498 at http://redmine.yorba.org/show_bug.cgi?id=4498
Unknown milestone "unknown in product shotwell. Setting to default milestone for this product, "---". Setting qa contact to the default for this product. This bug either had no qa contact or an invalid one.
Version: 0.15.0
Resolution: RESOLVED INVALID