Category Archives: Linux

Linux and Open Source

Old patch for Bonnie++ to use random data rather than 0′s

Posted on by

Way back in 2007 I posted a blog about testing ZFS/FUSE with Bonnie++ using random data rather than 0′s, and I said:

it’s not ready for production use as it isn’t controlled by a command line switch and relies on /dev/urandom existing. yes, I’m going to send the patch to Russell to look at

I didn’t get any feedback on the patch, so I’ve decided to post it here in case people are interested.

diff -ur bonnie++-1.03a/bonnie++.cpp bonnie++-1.03a-urand/bonnie++.cpp
--- bonnie++-1.03a/bonnie++.cpp 2002-12-04 00:40:35.000000000 +1100
+++ bonnie++-1.03a-urand/bonnie++.cpp   2007-01-01 13:03:41.644378000 +1100
@@ -41,6 +41,9 @@
 #include <string.h>
 #include <sys/utsname.h>
 #include <signal.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
 
 #ifdef AIX_MEM_SIZE
 #include <cf.h>
@@ -148,6 +151,28 @@
   }
 }
 
+void load_random_data(char *temp_buffer,int length)
+{
+       int filedes, numbytes;
+
+       filedes=open("/dev/urandom",O_RDONLY);
+       if(filedes<0)
+       {
+               perror("Open of /dev/urandom failed, falling back to 0's");
+               memset(temp_buffer, 0, length);
+       }
+       else
+       {
+               numbytes=read(filedes,temp_buffer,length);
+               if(numbytes!=length)
+                       {
+                               perror("Read from /dev/urandom failed, falling back to 0's");
+                               memset(temp_buffer, 0, length);
+                       }
+               close(filedes);
+       }
+}
+
 int main(int argc, char *argv[])
 {
   int    file_size = DefaultFileSize;
@@ -477,7 +502,8 @@
       return 1;
     globals.decrement_and_wait(FastWrite);
     if(!globals.quiet) fprintf(stderr, "Writing intelligently...");
-    memset(buf, 0, globals.chunk_size());
+    // memset(buf, 0, globals.chunk_size());
+    load_random_data(buf, globals.chunk_size());
     globals.timer.timestamp();
     bufindex = 0;
     // for the number of chunks of file data

Second beta release of Vacation 1.2.7.1

Posted on by

Vacation 1.2.7.1 beta2 is the second beta for the first bug fix only release in the 1.2.7 branch.

This release just fixes up some issues that Coverity revealed, none of which appeared to be harmful.

NB: If you compile with GCC 4.6 and see a complaint about orighdr being set but never used in rfc822.c don’t worry, its already fixed in git and will be in the first RC (assuming nothing bad is found in this version).

Please grab this beta release and test it and report any problems!

Recovering 5.25″ Floppy Disks

Posted on by

When I was at the University College of Wales, Aberystwyth they were just starting to supplement their serial terminals connected to X.25 PAD’s with some PC’s (Viglen I think), complete with hard disks and 5.25″ floppy drives. So I have had two boxes of 5.25″ floppies which dutifully came out to Australia with me when I emigrated from the UK back in 2002. These floppies are now well over 20 years old so I reckoned it would be about time to see whether they were still readable and, if so, what was on them. Labels like “Honeywell Backup Disk #1” only say so much.. ;-)

The first problem was that I didn’t have a 5.25″ floppy drive. Luckily my boss at VLSCI was able to lend me one. The second problem was I didn’t have a floppy cable with the 5.25″ connector on it. Fortunately Bernie at work had an old PC that was in bits which did have one, so I was able to borrow that. Then I found that the old Dell PC I was thinking of using had a really weird connector and wasn’t going to be that useful. My even older Olivetti Netstrada (a quad processor Pentium Pro monster) did have IDE, but the cables were going to be rather hard to get to as it has a bunch of SCSI drives and the cables for them were going to make it hard.

My final box was an VIA EPIA V box (originally from EverythingLinux back in 2003) which did have an easily accessibly floppy connector on the mother board, but only a single power connector for a drive. So it was either the internal IDE disk, or the floppy, but not both. I could have gone and bought a power splitter, but I thought I’d take the cheaper way and netboot it (the onboard ethernet chipset has PXE support) – it should be pretty easy.. Hah! :-) This is what the box looked like after some careful assembly..

Resurrecting 5.25" floppy disks - hardware

The first test was to see if the 5.25″ floppy disk drive worked. Luckily I had a floppy labelled as “system boot disk” and after some mucking around in the BIOS (it turns out you can set it to boot from floppy without having enabled the floppy controller, which results in it not booting from floppy and much cursing until you discover it) it booted first time – a 23 year old DOS boot disk complete with partly bilingual Welsh/English welcome screen from 1988!

UCW Aberyswyth 5.25" floppy system disk from 1988

This was very promising – the first disk had worked first time and a quick test of swapping it out for another and doing a “DIR” also worked. Now to get the data off these before they went to the great /dev/null in the sky..

My plan to recover the info was to netboot this machine as a Mythbuntu diskless front end box – Mythbuntu makes that easy to set up and with a little fiddling of the DHCP server to make sure it would only every try and serve this box, and do so with a static address, it worked. Or at least it would load the kernel. Which then complained that it couldn’t boot as it needed a CPU which had PAE support. :-(

This VIA EPIA V has a low power (5W) 533MHz VIA Eden CPU (appropriately the kernel detects it as a VIA Samuel 2) and whilst it is IA-32 it doesn’t have some of the newer features which are selected for Pentium class processors in the current Linux kernel.

Oh well, that’s fine, I knew I could use Debian instead, so I used the ltsp-build-client (creating an /etc/sysconfig/ltspdist file first containing the line VENDORDEF="Debian" first so it would use the right set of scripts), thus:

ltsp-build-client --chroot sid --mirror http://mirror.internode.on.net/pub/debian/ --dist sid --purge-chroot --arch i386 --accept-unsigned-packages

Problem was that all the various kernel command line options for specifying the NFS server for the root filesystem just didn’t seem to work, it would just sit there saying something like “Waiting for root filesystem” and eventually give up and drop me to a busybox shell prompt, and a simple cat of /proc/cmdline showed the options were being set correctly. A little more thought and an examination of the config file for the kernel showed that Debian doesn’t ship kernels with CONFIG_NFS_ROOT set, so it was never going to work. :-(

Whilst I could have rolled my own kernel I decided to instead have a look to see if I could find a Linux distro that included PXE booting as an option and a Google search for “linux distro pxe” turned up PLoP Linux as the first hit.

PLoP Linux is a small (75MB) distro aimed at data recovery operations that comes as an ISO, tar.gz or zip file for i586, x86-64 and (crucially for me) i486 processors. They even have a separate tar file for the PXE files. It was easy to set up and so I booted the PC with high hopes. Then I got the same error about the kernel requiring PAE support in the CPU that I got with Ubuntu. Whilst there was an i486 tar file there wasn’t an i486 PXE tar file! That was easily solved by grabbing the i486 ISO, doing a loopback mount of it and stealing the kernel and initrd.img files from it instead.

This time it booted, and I found that it had just what I was after – a Linux shell prompt, working networking, mtools (for mcopy and mdir) and (most importantly) ddrescue to let me create complete images of the floppies. I created a directory for each floppy disk and then did ddrescue /dev/fd0 floppy.img to make the image. I created another directory called Contents and from there did mcopy -vms a: (yes, I ordered the options that way deliberately) to copy all the files and subdirectories off complete with their last modification times (from 1987 and 1988 generally).

In all I was able to recover 20 of the 21 floppies with no errors at all, which amazed me as I was expecting them to have degraded over time (especially as one box was just a flimsy cardboard box). I was hoping to have original B source code from HoneyBoard (the bulletin board that Alan Cox and others I knew there wrote) and AberMUD from the Honeywell L66 but sadly that doesn’t appear to be the case. There are 3 B programs but one is just just 3 lines (calling a drl) and the other two appear to be two versions of some sort of shell which I didn’t write as the first has a btidy timestamp from April 1987, before I arrived at Aberystwyth.

The 21st disk was completely unreadable – the drive didn’t seem to want to acknowledge its existence and ddrescue couldn’t see anything as the floppy driver in the kernel couldn’t get the drive to provide any data. I might try ddrescue’s mode of copying data in reverse to see if that manages any better..

Google to acquire Motorola Mobility (Updated x1)

Posted on by

Very interesting news, especially given Motorola’s recent sabre rattling about going after patent victims^W income – hopefully this will put the end to that nonsense.

MOUNTAIN VIEW, Calif. & LIBERTYVILLE, Ill. – Aug. 15, 2011 – Google Inc. (NASDAQ: GOOG) and Motorola Mobility Holdings, Inc. (NYSE: MMI) today announced that they have entered into a definitive agreement under which Google will acquire Motorola Mobility for $40.00 per share in cash, or a total of about $12.5 billion, a premium of 63% to the closing price of Motorola Mobility shares on Friday, August 12, 2011. The transaction was unanimously approved by the boards of directors of both companies.

The acquisition of Motorola Mobility, a dedicated Android partner, will enable Google to supercharge the Android ecosystem and will enhance competition in mobile computing. Motorola Mobility will remain a licensee of Android and Android will remain open. Google will run Motorola Mobility as a separate business.

I hope with Google in control we’ll see some better Android devices out there – can we get a real keyboard please ?!?

Update 1:

It appears that patents are part of the reason for Google buying Motorola, but looks like they’re being trailed as defensive according to this TechCrunch article:

During today’s conference call explaining the deal, Page noted that Motorola’s “strong patent portfolio” will help Google defend Android against “Microsoft, Apple, and other companies.” The first two questions on the call went right to the patent issue as well. With Android under attack on the patent front by Apple, Microsoft, Oracle and others, buying Motorola is very much a defensive move as well.

GNU/Linux based phones – sic transit gloria mundi

Posted on by

So it’s looking like there’s going to be a real dearth of “real” GNU/Linux based phones from now on, with the Nokia N9 likely to be the last that I can see for some time. For a while it was looking quite promising with the whole Meego idea, but Nokia’s new CEO from Microsoft seems to have put paid to that with their colours now tied to the sinking mast of Windows Mobile. On the Meego handset list changes to how bugzilla items relating to the handset CE were announced, with the disappointing message that:

Handset UX isn’t fully maintained and currently there’s no activities for 1.3. N900 CE project is only who is maintaining / developing anything to Handset UX and when bugs are fixed in CE the fixes are also available to add to the MeeGo release.

There has been some activity from Nokia contributing back some work done on Meego (in general) for the N9, but basically it appears there’s no interest in Meego phones from anyone other than the N900 community. :-(

That means that for now we’re stuck with either pseudo-open Android/Linux phones (which, I’ve got to say, haven’t really impressed me too much so far) and Apple’s iPhone which I dislike for its extreme walled garden approach (and I’ve even less idea about usability than Android).

I guess what I’m missing is the power of having a real Linux distro (a shell and utilities that you expect with it) on a phone. I’ve been really happy with my N900 – the fact that I can ssh to our HPC clusters and use X forwarding to fire apps back to it has been really useful at times (not to mention vendor provided root access) – and I was struck that the Meego handset UX has full bash and (amazingly) even strace installed by default. Something approaching a real computer.

But then again I’m a niche market that makes most niche markets look positively huge, but having tasted freedom with the OpenMoko and the N900 I’m hungry for more..

SOLVED: Modern kernels fail to boot on old quad processor Pentium Pro server (Updated x3)

Posted on by

I’ve got an ancient Olivetti Netstrada, a deskside server system with quad Pentium Pro 200MHz processors, 256MB RAM, dual power supplies and five 4GB SCSI drives.

It’s been running Ubuntu 8.04 for ages and I found that with my partitioning layout (set up for testing ZFS-fuse ages ago) I couldn’t upgrade it without major surgery so I decided I’d just put Debian on it instead. That’s where I hit problems..

  1. Debian/kfreebsd (Squeeze & daily) – kernel panics very early with panic: vm_fault: fault on nofault entry, addr: c3925000.
  2. Debian/Linux Squeeze – CD boot loader hangs before getting to menu
  3. Debian Lenny – install kernel panics when uncompressing the initramfs, claims it’s out of memory.

Fortunately the Debian Etch install CD boots and installs correctly, only problem is that Etch is now archived and there are no updates for it..

I dist-upgraded to Lenny and found that the latest kernel there still panics on boot, but the user space is OK. Then I went to Squeeze and found that yes, the Squeeze kernel hangs very early, just after saying it was booting the kernel after uncompressing. Unfortunately the udev in Squeeze won’t work with the Etch kernel, but all that’s broken so far is bringing up the network interface and I can do that by hand with dhclient eth0. Oh, and grub2 hangs (which I suspect is the same issue as the install CD).

I’ve tried building my own kernel using 2.6.38.3 starting with an “allnoconfig” to disable everything and only turn on the minimum necessary, but that has the same behaviour as the 2.6.32 kernel that is in Squeeze, the last thing printed to the console is:

Booting the kernel.

which is at the end of the decompress_kernel() function in arch/x86/boot/compressed/misc.c.

Does anyone have any ideas before I go and throw myself on the tender mercies of the LKML ?

Update: Both Alan Cox and Ingo Molnar suggested using the earlyprintk=vga option which I’d not stumbled across before, that revealed that the 2.6.39-rc4 kernel is misdetecting LOWMEM as 16MB not 256MB which could explain a lot. It also reminded me that I’d seen this before and had an offlist conversation with H. Peter Anvin about it in 2008 which tailed off due to work pressures on his part.

Update 2: Thanks to Thomas Meyer and H. Peter Anvin it’s now known what happened – the commit message from hpa for Thomas’s patch describes it best:

When we use BIOS function e801 to probe memory, we should use ax/bx (or cx/dx) as a pair, not mix and match. This was a typo during the translation from assembly code, and breaks at least one set of machines in the field (which return cx = dx = 0).

The patch has been accepted by Linus and will be in 2.6.39!

Update 3: The patch is in 2.6.39-rc6 and that now successfully boots all the way to userspace with the kernel parameters “noapic scsi_mod.scan=sync”! Hooray!

NUMA, memory binding and how swap can dilute your locality

Posted on by

On the hwloc-devel mailing list Jeff Sqyres from Cisco posted a message saying:

Someone just made a fairly disturbing statement to me in an Open MPI bug ticket: if you bind some memory to a particular NUMA node, and that memory later gets paged out, then it loses its memory binding information — meaning that it can effectively get paged back in at any physical location. Possibly even on a different NUMA node.

Now this sparked off an interesting thread on the issue, with the best explanation for it being provided by David Singleton from ANU/NCI:

Unless it has changed very recently, Linux swapin_readahead is the main culprit in messing with NUMA locality on that platform. Faulting a single page causes 8 or 16 or whatever contiguous pages to be read from swap. An arbitrary contiguous range of pages in swap may not even come from the same process far less the same NUMA node. My understanding is that since there is no NUMA info with the swap entry, the only policy that can be applied to is that of the faulting vma in the faulting process. The faulted page will have the desired NUMA placement but possibly not the rest. So swapping mixes different process’ NUMA policies leading to a “NUMA diffusion process”.

So when your page gets swapped back in it will drag in a heap of pages that may have nothing to do with it and hence their pages may be misplaced. Or worse still, your page could be one of the bunch dragged back in when a process on a different NUMA node swaps something back in. This will dilute the locality of the pages. Not fun!

How computers fail to entice people into being programmers

Posted on by

An old friend of mine from the UK, Steve Usher, has pretty much nailed things with this blog on “Enthusing teen minds: Why today’s computers won’t create tomorrow’s programmers“. He says:

The computers of the early 80s were a blank canvas. You plugged them in, switched them on and (hopefully) the input cursor blinked at you. There was no decoration, no clutter and it was something waiting for YOU to do something to it.

Ah yes, I remember those days, when 3.5KB was a lot of memory! But what about today’s computers ?

They’re immediately brimming full of functionality all vying for your attention, but it’s also incredibly locked down. You can do absolutely anything… ANYTHING as long as it’s what the visionary who steered the programming teams thinks that you should want to do. Woe betide you if you want to do anything different. It’ll either ignore you or give you an unhelpful suggestion in a dialog box. You can be creative, but only in the ways you’re told you can be.

But before us free software types get all puffy and “I told you so”, he points out that things aren’t that much better on our systems with all our SDK’s, IDE’s, toolkits, compilers and interpreters:

It’s like taking a 5 year old into an engineering workshop, sitting him down and then complaining when he doesn’t build a car as he had all the tools available to him to do it and hence it must be his fault.

I’m not as sure that we need to build something new from scratch though, I think it might be more the case that what we need to do is to sort through all the various projects that could fit what he is after and build a distro (of whatever OS) that boots up straight into that application and lets them play with it. Perhaps something like SDLbasic (a BASIC interpreter for game development) might be a good start ?

Testing the N900 “Lowlight” Photo Application

Posted on by

Nokia have released some “research prototypes” of applications that use the FCam libraries to do fun things with taking photos on their Nokia N900. One of these is called “Lowlight” and is designed to make it easier to get reasonable photos in, well, low light conditions. I tested this out on Tuesday night after the Linux Users of Victoria meeting, taking a photo of the Old Commerce Building on the University of Melbourne campus.

Old Commerce building, University of Melbourne, at night

Now given that’s taken hand-held without a flash, I’m pretty impressed, you can even make out the design of the courtyard floor in front of it! If you go and look at the original full size version on Flickr (CC-BY licensed) you can see there is noise around the outside of the building, and there is an almost oil-paint effect on the details of the carvings on the building due to their algorithm, but given the alternative was nothing at all it’s a great little program!

VLSCI: Systems Administrator – High Performance Computing, Storage & Infrastructure

Posted on by

/*
* Please note: enquiry and application information via URL below, no agencies please!
*
* Must be Australian permanent resident or citizen.
*/

Executive summary

Want to work with hundreds of TB of storage, HPC clusters and a Blue Gene supercomputer and have an aptitude for handling storage and data ?

http://jobs.unimelb.edu.au/jobDetails.asp?sJobIDs=715542

Background

VLSCI currently has in production as stage 1:

  • 2048 node, 8192 core IBM Blue Gene/P supercomputer
  • 80 node, 640 core IBM iDataplex cluster (Intel Nehalem CPUs)
  • ~300TB usable of DDN based IBM GPFS storage plus tape libraries
  • 136 node, 1088 core SGI Altix XE cluster (Intel Nehalem CPUs)
  • ~110TB usable of Panasas storage

There is a refresh to a much larger HPC installation planned for 2012.

Both Intel clusters are CentOS/RHEL 5, the front end and service nodes for the Blue Gene are SuSE SLES 10. The GPFS servers are RHEL5. Panasas runs FreeBSD under the covers.

Job advert

http://jobs.unimelb.edu.au/jobDetails.asp?sJobIDs=715542

SYSTEMS ADMINISTRATOR, STORAGE & INFRASTRUCTURE
Position no.: 0022139
Employment type: Full-time Fixed Term
Campus: Parkville

Close date: February 3rd, 2011

Victorian Life Sciences Computation Initiative, Melbourne Research

Salary: HEW 7: $69,608 – $75,350 p.a. or HEW 8: $78,313 – $84,765 p.a. plus 17% superannuation.

The Victorian Life Sciences Computation Initiative (VLSCI) is a Victorian Government project hosted at The University of Melbourne which aims to establish a world class Life Sciences Compute Facility for Victorian researchers. The Facility operates a number of supercomputers and storage systems dedicated to High Performance Computing. The VLSCI wishes to recruit a Linux Systems Administrator with knowledge of file systems and an interest in working with technologies such as GPFS, TSM, HSM, NFS.

This position is an opportunity to become involved in leading science and computing fields and work as part of a small but self-contained team. Expect to find yourself learning new skills and developing new and innovative solutions to problems that have not yet been identified. You have every opportunity to make a real difference and will need to contribute to a high level of service and creativity.

More Details

Selection criteria and more details are in the Position Description (PDF) here:

http://bit.ly/fyIy3H

Apologies for the URL shortener, the original URL is a horribly long one.. :-(