libraw-config can select non-thread-safe libraw
Submitted by an unknown user
Assigned to Jim Nelson
Link to original bug (#716874)
Description
---- Reported by shotwell-maint@gnome.bugs 2010-09-07 14:49:00 -0700 ----
Original Redmine bug id: 2528
Original URL: http://redmine.yorba.org/issues/2528
Searchable id: yorba-bug-2528
Original author: Valentin David
Original description:
The bug might be related to #2514 (closed).
The version tested is the svn r2197.
I cannot import my big library correctly, because I get hardware errors even though my disk is intact and the files are correct. I cannot also properly use my computer during the import which last several hours.
The reason is that importing reads 4 different files in the same time. I do not expect any program to do that.
Either there should be a file system driver as proxy that has locks input output operations, which would be ideal. Eventually with caching, see mmap.
Or reduce the number of threads.
As a workaround I made a LD_PRELOAD that adds mutex the syscalls read and write (given that my libglib.so use dynamic symbols for them from libc instead of doing the syscall). It seems to fix the problem (i.e. not hanging my other processes needing disk access), while still using all the processor resource (load average more than 1).
---- Additional Comments From shotwell-maint@gnome.bugs 2013-05-01 11:40:00 -0700 ----
History
Comment 1
Updated by Valentin David about 3 years ago
Actually the workaround does not work. Locking is useless as the system calls already do it. It seems the problem arise when the files are big enough (basically raw files). However forcing the load the whole file and locking that could routine help, as we do not want a call to system call read to a file before we did not finish loading all the other files.
Comment 2
Updated by Jim Nelson about 3 years ago
- Priority changed from Urgent to High
I'm curious what errors you're seeing -- what exactly does Shotwell report when you import these files?
I'm also curious what kind of system you're running on -- CPU, hard drive, number of files, distro, etc.
Opening four files at once is not an odious task for the operating system or unusual for modern applications. RAW files are difficult to process and loading them can cause a performance hit on the CPU. We can throttle the workload so that it doesn't hit the system so hard, but that would (most likely) come at a cost of time it takes to import the photos.
We reserve critical tickets for crashers and showstoppers. I'm moving this down to high, as it might be something we address in 0.8.
Comment 3
Updated by Valentin David about 3 years ago
The lines are like that:
L 10110 2010-09-08 00:15:18 [%(=caps)WRN%] Photo.vala:671: Unable to interrogate photo file /home/lgv/MyPictures/2007/08/04/IMG_2848.CR2: unpack: Input/output error
Several of them. But not the same files when launching again from scratch. This is why I believe it is not my filesystem. The files are also good and can be read without errors. Only raw files. Never got it on jpeg files. But I get it quite often on raw files.
I took that “Input/output error†was from some sys call errors. But now I think it might actually be some thread problems.
I am wondering if it is linked to the right libraw file. There might be one that is thread-safe and not the other.
Comment 4
Updated by Valentin David about 3 years ago
- Status changed from Open to 5
- Resolution set to fixed
- % Done set to 100
My bad, I think I did something wrong. However, I still think that the importing is really taking too much my disk because it is several threads reading the disk, which makes other programs to just hang for a while.
Comment 5
Updated by Jim Nelson about 3 years ago
First, there definitely are two versions of libraw, one thread-safe and the other not. We link specifically to libraw_r, which is the thread-safe version. If you've built it yourself, you should be sure the thread-safe version is installed.
It looks like the error message you're seeing is from libraw. Without investigating too deeply, it looks like while it was decompressing/decoding the RAW file it hit an I/O error.
As I said before, import performance is a major concern for us, and we'll consider ways we can reduce the workload while keeping in mind total time to import.
Comment 6
Updated by Valentin David about 3 years ago
Thank you. For information I succeeded to make it work properly. Though it is not finished importing my library, it has been working all day without an error. Probably I will see tomorrow morning if everything worked alright.
But I had to modify your script libraw-config. I use a Gentoo Linux, and it has two pkg-config files: libraw.pc and libraw_r.pc. It is specific to this distribution as it is a patch from the Gentoo portage that does this. Your script tried the first one and got linked to -lraw instead of -lraw_r. I suggest you check for pkg-confg --exists libraw_r first and use libraw if it was not found. Or eventually check that what pkg-config did not give you -lraw when -lraw_r is available.
It was not very obvious. And it has been many hours of processing that had to be redone. That would be nice that this problem does not happen to nobody else. Specially that the version of shotwell on Gentoo in the portage is not updated, so it is better to compile it from source.
Comment 7
Updated by Jim Nelson about 3 years ago
- Status changed from 5 to 4
-
Resolution deleted (
<strike>
_fixed_</strike>
) - % Done changed from 100 to 0
-
Priority deleted (
<strike>
_High_</strike>
) - Subject changed from Shotwell should not read multiple files in several threads on the same disk to libraw-config can select non-thread-safe libraw
That explains a lot. libraw-config should never use the non-thread-safe libraw. Bad news indeed.
Can you send me a diff of your changes to the libraw-config file? I would appreciate that.
I've re-opened this (with a different title) to reflect this problem.
Comment 8
Updated by Jim Nelson about 3 years ago
- Status changed from 4 to Review
- Assignee changed from Anonymous to Jim Nelson
Comment 9
Updated by Valentin David about 3 years ago
I gave you the patch as you asked. However, you should do the same thing with the cflags function.
Comment 10
Updated by Jim Nelson about 3 years ago
- Status changed from Review to 5
- Resolution set to fixed
- % Done changed from 0 to 100
Thanks for the patch. I've committed it to branch and should soon appear in trunk.
Comment 11
Updated by Charles Lindsay 7 months ago
- Status changed from 5 to Fixed
--- Bug imported by chaz@yorba.org 2013-11-25 21:47 UTC ---
This bug was previously known as bug 2528 at http://redmine.yorba.org/show_bug.cgi?id=2528 Imported an attachment (id=261831)
Unknown Component Using default product and component set in Parameters Unknown version " in product shotwell. Setting version to "!unspecified". Unknown milestone "unknown in product shotwell. Setting to default milestone for this product, "---". Setting qa contact to the default for this product. This bug either had no qa contact or an invalid one.
Resolution: RESOLVED FIXED