Occasionally, with some raw+jpeg images that contain EXIF segments longer than 64 kbytes, the sister .jpg will get imported separately as well.
Submitted by cli..@..ba.org
Assigned to cli..@..ba.org
Link to original bug (#719060)
Description
---- Reported by clinton@yorba.org 2013-01-31 12:19:00 -0800 ----
Original Redmine bug id: 6292
Original URL: http://redmine.yorba.org/issues/6292
Searchable id: yorba-bug-6292
Original author: Clinton Rogers
Original description:
This was observed after the following:
- With a directory containing at least ten raw+jpeg pairs and an empty/fresh database and image library directory, launch Shotwell.
- Via File->Import From Folder, import the directory from step one.
On rare occasions, even though the sister .jpgs should be seen as part of a pair and blacklisted from being imported, one or two of them will get imported twice, once as part of a pair, and a second time as an independent image.
Although serious, this has only been seen twice by me and, to my knowledge, once by another Yorban; further attempts to reproduce it haven't been successful.
Related issues:
- related to shotwell - 6202: Development files with .jpg
<pid>
extension left on filesy... (Blocked)
---- Additional Comments From shotwell-maint@gnome.bugs 2013-05-01 11:47:00 -0700 ----
History
Comment 1
Updated by Jim Nelson 10 months ago
- Category changed from import to raw
- Priority changed from Low to Normal
Comment 2
Updated by Clinton Rogers 10 months ago
I just saw this again, but ONLY with the .srw files (this is in a large batch import of mixed raw filetypes).
Comment 3
Updated by Jim Nelson 10 months ago
- Target version changed from 0.15.0 to 0.14.0
This is the second serious problem we've seen with SRW's. Let's see if we can get this in for 0.14.
Comment 4
Updated by Clinton Rogers 10 months ago
I am also seeing the '_embedded' and '_shotwell' developments get added to the DB (again, only for .SRW files); something about the way these are handled is breaking horribly with the import blacklisting mechanism.
epilogue:
Last week's research provided insight into the matter: it seems that many
Samsung cameras produce images with EXIF segments longer than 64 kbytes (a
spec violation), and the library we use to write metadata out to images,
libexiv2, fails when asked to write a non-compliant EXIF segment into an
image. Since, when making a development JPEG, we get the EXIF data directly
from the source image, we were ending up with unwriteable EXIF segments, and
Shotwell was erroring out while creating the developments (and, as such,
thought it needed to create another one (or more)).
I believe this should be greatly improved by work done for #6390 and #5681, although with some pathological raw developments, it's most likely still possible (the metadata won't get written, and as such won't match the source image, and so Shotwell incorrectly sees it as a new image).
Comment 5
Updated by Jim Nelson 9 months ago
- Target version changed from 0.14.0 to 0.15.0
Comment 6
Updated by Jim Nelson 9 months ago
- Assignee set to Clinton Rogers
We need to work with the Exiv2 folks to get this fixed. It does seem like that's the right way to solve this problem.
Comment 7
Updated by Joe Bylund 9 months ago
Actually I've seen similar behavior with both nikon nef and olympus orf files. Where the raw will be imported and when I later look at my files I see I have both the raw file and the _embedded or _shotwell file. So 1 file import leads to 2 in the db. I don't know if I've ever seen this while doing a small import.
For my most recent event, for which I don't think I removed and _embedded files, I have 7 in the db (5 nikon, 2 olympus), everything was shot in raw. (totals of ~158 olympus, ~276 nikon photos, only approx because of the _embedded imports and having edited some by hand and doubly imported as jpg).
Comment 8
Updated by Clinton Rogers 9 months ago
Joe: thank you for taking time to alert us to this.
If you feel comfortable doing so, would you be willing to let us take a look at two of the files in question (one Olympus, one Nikon)?
Also, for the files that this happened with, did you see any stray files left over named WHATEVER_THE_FILENAME_WAS.jpgXXXXX in the library directory?
Comment 9
Updated by Joe Bylund 9 months ago
If you feel comfortable doing so, would you be willing to let us take a look at two of the files in question (one Olympus, one Nikon)?
Files are over the size limit (9.6M & 12M). Do you guys have somewhere I can drop the file such that it'd be web accessible to everyone, or maybe you can increase the size limit a bit?
In the database the jpgs are named _DSC7184_NEF_embedded.jpg & P2102781_ORF_embedded.jpg
Also, for the files that this happened with, did you see any stray files left over named WHATEVER_THE_FILENAME_WAS.jpgXXXXX in the library directory?
I've never seen that sort of file, so no guarantee's this is caused by the same thing.
Next time I do a big import and see this I'll grab the log.
Comment 10
Updated by Jim Nelson 9 months ago
Joe, if you put them on a file sharing service or on a Dropbox link, you can email that link to geary@yorba.org and one of us will pick it up.
Comment 11
Updated by Joe Bylund 9 months ago
Here ya go:
Olympus: http://ubuntuone.com/4UCKSGm7yc1vP1c1mb3ug9
Nikon: http://ubuntuone.com/0wJ0Nu76aBmdtuaGfTzXm5
Comment 12
Updated by Clinton Rogers 9 months ago
Thank you so much for this. I have them now, so if you'd rather not share them for too long, they're safe to take down now.
Comment 13
Updated by Joe Bylund 9 months ago
I think I forgot to mention earlier. I usually keep filesystem monitoring turned on, and I have a suspicion that this is due to filesystem monitoring picking up the development before it gets imported. I guess a workaround would be to temporarily toggle off filesystem monitoring during import? Or at least it would be an interesting test.
I'll leave them up as long as space permits.
Comment 14
Updated by Clinton Rogers 9 months ago
What's supposed to happen is that the development image is supposed to be 'blacklisted' (in the parlance of the code) during import, but for some reason, the blacklist mechanism isn't working.
I just tried importing these with a very small import, and it works fine then, so import size is definitely part of what triggers this.
Comment 15
Updated by Clinton Rogers 9 months ago
- % Done changed from 0 to 10
No stray .JPG<pid>
files were created (please see #6202), so this may be a
different failure mode.
Comment 16
Updated by Clinton Rogers 9 months ago
- Subject changed from [norepro] Rarely, when importing multiple raw+jpeg pairs via Import From Folder, a sister .jpg will get imported separately as well. to Rarely, when importing multiple raw+jpeg pairs via Import From Folder, a sister .jpg will get imported separately as well.
Comment 17
Updated by Clinton Rogers 8 months ago
- % Done changed from 10 to 30
Confirmed that this seems to be more likely to occur with larger imports and slower media; with a 1000-file import from an SDHC card in a relatively recent reader, no luck, but on a 750-file import from an old USB 1.x media player, I've triggered it twice.
Comment 18
Updated by Clinton Rogers 8 months ago
Breaking the slow USB media problem out into its own bug.
Comment 19
Updated by Clinton Rogers 8 months ago
- Subject changed from Rarely, when importing multiple raw+jpeg pairs via Import From Folder, a sister .jpg will get imported separately as well. to Occasionally, with some raw+jpeg images that contain EXIF segments longer than 64 kbytes, the sister .jpg will get imported separately as well.
Comment 20
Updated by Clinton Rogers 8 months ago
- % Done changed from 30 to 60
This does not appear to be reproducible with libexiv2 0.23 (Raring). Precise and Quantal will, by default, install 0.22.
Comment 21
Updated by Clinton Rogers 8 months ago
- Status changed from Open to 5
- % Done changed from 60 to 100
- Resolution set to worksforme
On Quantal, 0.23 is available through updates.
Unless otherwise directed, I'd like to close this on the grounds that several known-to-be-pathological-with-0.22 files work as intended in 0.23,
For users that encounter this, it may be helpful to upgrade libexiv2 to 0.23 or higher.
Comment 22
Updated by Charles Lindsay 7 months ago
- Status changed from 5 to Invalid
--- Bug imported by chaz@yorba.org 2013-11-25 21:59 UTC ---
This bug was previously known as bug 6292 at http://redmine.yorba.org/show_bug.cgi?id=6292
Unknown milestone "unknown in product shotwell. Setting to default milestone for this product, "---". Setting qa contact to the default for this product. This bug either had no qa contact or an invalid one.
Version: 0.15.0
Resolution: RESOLVED INVALID