Speeding up Digikam’s face recognition (with risks)

It’d been a while since I’d last told Digikam to scan my collection for faces, and having just upgraded to 3.2.0 I thought it was about time to have another shot at it. However, I’d noticed it was taking an awful long time and seemed to only be using one of the eight cores on this system (Ivy Bridge i7-3770K running Kubuntu 13.04) so I thought I’d see if simply taking advantage of OpenMP could improve things with multithreading.

To do that I just started a new konsole and (as a first step) told OpenMP to use all the cores with:

export OMP_NUM_THREADS=8

Running digikam from that session and starting a face scan showed that yes, it was using all 8 cores, but not really to a great amount. Running iotop showed it doing about 5MB/s in reads and latencytop showed that it was spending most of its time in fsync(). Now that’s good, because it’s making sure that the data has really hit the rust to ensure everything is consistent.

However, in this case I can rebuild the entire face database should I need to, and I have about 66GB of photos to scan, plus I wanted to see just how fast this could go. ;-) So now it’s time to get a little dangerous and try Stewart Smith’s wonderfully named “libeatmydata” library which gives you a library (surprise surprise) and helper program that lets you preload an fsync() function that really only does return(0); (which, you may be interested to know, is still POSIX compliant).

So to test that out I just needed to do:

eatmydata digikam

and suddenly I had 8 cores running flat out. iotop showed that Digikam was now doing about 25-30MB/s reads and latencytop showed most of its time waiting for things was now for user space lock contention, i.e. locks protecting shared data structures to stop threads from stomping on each other and going off into the weeds. Interestingly the disks are a lot quieter than before too. Oh, and it’s screaming through the photos now. :-)

WARNING: Do not use eatmydata for anything you care about, it will do just what it says in the name should your power die, system hang, universe end, etc..

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>