Speeding up Digikam’s face recognition (with risks)

It’d been a while since I’d last told Digikam to scan my collection for faces, and having just upgraded to 3.2.0 I thought it was about time to have another shot at it. However, I’d noticed it was taking an awful long time and seemed to only be using one of the eight cores on this system (Ivy Bridge i7-3770K running Kubuntu 13.04) so I thought I’d see if simply taking advantage of OpenMP could improve things with multithreading.

To do that I just started a new konsole and (as a first step) told OpenMP to use all the cores with:

export OMP_NUM_THREADS=8

Running digikam from that session and starting a face scan showed that yes, it was using all 8 cores, but not really to a great amount. Running iotop showed it doing about 5MB/s in reads and latencytop showed that it was spending most of its time in fsync(). Now that’s good, because it’s making sure that the data has really hit the rust to ensure everything is consistent.

However, in this case I can rebuild the entire face database should I need to, and I have about 66GB of photos to scan, plus I wanted to see just how fast this could go. 😉 So now it’s time to get a little dangerous and try Stewart Smith’s wonderfully named “libeatmydata” library which gives you a library (surprise surprise) and helper program that lets you preload an fsync() function that really only does return(0); (which, you may be interested to know, is still POSIX compliant).

So to test that out I just needed to do:

eatmydata digikam

and suddenly I had 8 cores running flat out. iotop showed that Digikam was now doing about 25-30MB/s reads and latencytop showed most of its time waiting for things was now for user space lock contention, i.e. locks protecting shared data structures to stop threads from stomping on each other and going off into the weeds. Interestingly the disks are a lot quieter than before too. Oh, and it’s screaming through the photos now. 🙂

WARNING: Do not use eatmydata for anything you care about, it will do just what it says in the name should your power die, system hang, universe end, etc..

Problems getting stack traces from a Python program (Kubuntu 12.10 development version)

I’m trying to get to the bottom of this bug on Launchpad which completely breaks Synaptic touchpad configuration under KDE:

https://bugs.launchpad.net/ubuntu/+source/synaptiks/+bug/1039261

The tl;dr version is that the Python interpreter is somehow emitting two calls to the Xorg libXi function XIQueryVersion(), the first call sends a client XInput version number of 2.1 and then the second one sends 2.0 (seen using xtrace).

The second call causes a BadValue error, because you’re not meant to send a lower value on any later calls (as can be seen from this Xorg libXi git commit).

This causes the comical error:

The version of the XInput extension installed on your system is too old. Version 2.0 was found, but at least version 2.0 is required

The problem is that the Python code only has the second call sending the 2.0 version number, there is no other call in the package that will send anything else, let alone the 2.1 value.

So I want to generate a call trace every time the XIQueryVersion() function is called, but I’m struggling to get it to work.

The killer at the moment is that both ltrace and gdb (when told to trace children) hang when python runs dash to run ldconfig.real and that blocks – so I never get to the point where the function gets called the first time.

With GDB I’m using:

set detach-on-fork off
set follow-fork-mode child
set follow-exec-mode new
catch load /libXi/
break XIQueryVersion

…and this is what happens:

chris@chris-ultralap:~/Code/Ubuntu$ gdb /usr/bin/python
GNU gdb (GDB) 7.5-ubuntu
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/bin/python...Reading symbols from /usr/lib/debug/usr/bin/python2.7...done.
done.
(gdb) set detach-on-fork off
(gdb) set follow-fork-mode child
(gdb) set follow-exec-mode new
(gdb) catch load /libXi/
Catchpoint 1 (load)
(gdb) break XIQueryVersion
Function "XIQueryVersion" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 2 (XIQueryVersion) pending.
(gdb) run /usr/bin/synaptiks
Starting program: /usr/bin/python /usr/bin/synaptiks
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New process 3788]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Thread 0x7ffff6ccc700 (LWP 3788) is executing new program: /bin/dash
[New process 3789]
process 3789 is executing new program: /bin/dash
process 3789 is executing new program: /sbin/ldconfig.real

…and there it hangs, forever. We never even get to the point where the Python interpreter loads libXi.so, let alone calls the function. 🙁

Any ideas?

Dependency Problems with KDE 4.6.0 RC1 Packages for Ubuntu 10.10 (FIXED)

Update: this issue has been fixed by the packagers. See the bug for more info.

I saw the notice about the KDE 4.6.0 RC1 packages being released for Kubuntu 10.10 today and decided to look at upgrading (or at least grab all the packages with aptitude dist-upgrade -d) and got a surprise that it wanted to remove 250 packages due to dependency problems. The issues were:

The following packages have unmet dependencies:
   libqapt-runtime: Depends: libpolkit-qt-1-1 which is a virtual package.
   polkit-kde-1: Depends: libpolkit-qt-1-1 which is a virtual package.
   kdelibs5-plugins: Depends: libpolkit-qt-1-1 which is a virtual package.
   libgpgme++2: Depends: libgpg-error0 (>= 1.10) but 1.6-1ubuntu2 is installed.

Turns out that libpolkit-qt-1-1 and the newer version of libgpg-error0 are in the alpha for the next version of Kubuntu (Natty Narwhal) but not in Maverick (10.10), hence the chaos. The bug is reported as LP bug #694053 and thanks to Murz for spotting that you can just grab and dpkg -i the missing packages – see the bug report for details and links!

Bugs in KDE 4.5.0 beta 2

If you’re running KDE 4.5.0 beta 2 from the Kubuntu PPA and are wondering why your widgets disappear every time you login, it’s because beta 2 has a bug where it gets really confused about which activity you are using – use meta-tab (windows-tab) to switch between them and you should eventually find what you’re looking for..

Also if you’re wondering why the “Printer Settings” option isn’t working any more and giving a strange error message, it’s because it’s now looking for a Python module which isn’t packaged in Ubuntu (kubuntu-ppa bug 591980), it’s complaining that:

systemsettings(22750)/python (plugin): Failed to import module
 Traceback (most recent call last):
   File "/usr/share/kde4/apps/system-config-printer-kde/system-config-printer-kde.py", line 64, in 
     import cupsutils.ppds
 ImportError: No module named cupsutils.ppds
 systemsettings(22750)/python (plugin): Failed to import module

In Debian the module it’s looking for is in the “python-cupsutils” package, but that’s not in Ubuntu.

Kubuntu Users – Shave a Few Seconds Off Your Boot Time

If you’re a KDE Ubuntu user (Kubuntu) you might not know that the standard readahead package (which attempts to speed up boot by preloading the kernel file cache by efficiently reading them in advance) is tuned for the Gnome version of Ubuntu and so will be trying to read a bunch of libraries that are probably not installed on your system. So it’s unlikely to do much for your average KDE user!

However, if when you boot you go into the Grub boot loader and add the “profile” option to the end of your command line (after “quiet” for instance) then it will profile the files used during the startup and create new files in /etc/readahead so that next time you start you’ll actually be preloading the files that will be used on your system. I’ve found it’s shaved over 2.5 seconds off my boot time as recorded by bootchart.

There is a bug in Launchpad for this, it’s 369506 but it hasn’t had a lot of love. 🙁

I Can Haz Android (on an OpenMoko Freerunner)

I have been assimilated. Or at least my OpenMoko Freerunner has been! It’s now running the Koolu port of Android 1.5 “Cupcake” on it, and with a *very* helpful hint from Damian Spriggs on the OpenMoko community mailing list it’s able to make and receive calls and SMS’s. For the record you need to get ADB working and grab a root shell on the phone. Then you can use the sqlite command line utility to set the “provisioned” flag in its DB.

# sqlite3 /data/data/com.android.providers.settings/databases/settings.db
SQLite version 3.5.9
Enter ".help" for instructions
sqlite> INSERT INTO secure (name, value) VALUES ('device_provisioned', 1);

I’ve also found a rather nice application called VCardIO for importing my contacts exported in VCard v2.1 format from KDE’s Kontact addressbook. Now we’ll see how it goes over the next few days!

ODF Plugfest

After the noise over whether or not the implementation of ODF (Open Document Format) in SP2 for Microsoft Office 2007 was deliberately broken for monopolistic purposes or incompetently implemented (or a combination of both) it’s nice to see that there is active interoperability work going on between vendors and developers at the ODF PlugFest, and the KOffice developers Jos van den Oever and Sven Langkamp attended and contributed to an article on the KDE DOT news website and Sven blogged about his positive experiences at the workshop.

It was first time I was going to such a workshop and I had expected that there would be fights between the different vendors like it happened in some blogs before the workshop. It was a pleasant surprise for me that the athmosphere was very friendly and productive. It was really nice to meet other people projects/companys, put the competition aside for some time, work and drink some beer together.

One neat feature mentioned there is the OfficeShots website which lets you submit an ODF document and then get back renderings of it (PDF, screenshot, ODF) from various ODF implementations. There are 8 listed at present (including KOffice), but sadly MS Word or Google Docs aren’t amongst them (yet).

Digikam Problem with KDE 4.3 beta 1

If you’re using the Kubuntu 4.3 beta 1 packages for Kubuntu Jaunty you’ll likely find that you suddenly can’t view your albums any more. I traced it down to the following error in my ~/.xsession-errors file:

Could not open library '/usr/lib/kde4/kio_digikamalbums.so'.
Cannot load library /usr/lib/kde4/kio_digikamalbums.so: (/usr/lib/libdigikamcore.so.1: undefined symbol: _ZN6Marble12MarbleWidget16addPlaceMarkDataERK7QStringS3_)

Basically Digikam needs recompiling against the version of Marble in KDE 4.3 to get its symbol names fixed. Doing it isn’t that hard, you need to do:

sudo apt-get build-dep digikam
apt-get source digikam
cd digikam-0.10.0
dpkg-buildpackage

Once that’s done (and it’ll take a while) you’ll have 3 new packages in the parent directory of digikam-0.10.0, the Digikam package, a package for showfoto and a package for Digikams debugging symbols. Just use “sudo dpkg -i ” to install them.