Well, since, I got Debian installed on my GP2X, the first thing I did was run some benchmarks. As per usual, I fell back on my “raw and simple math computation” benchmark suite Primes, a collection of prime number finders written in many languages. For the rest of the article we will thus be evaluating languages purely on math computational speed and nothing else.

I had already gathered data from some of my computers from when I bought Bast, my G3, so I just ran the benchmarks on the GP2X and added them in for comparison.

Machines

Name Arch Speed OS
Inferno x86 – Athlon 1500MHz Getnoo Linux
Nika x86 – Pentium-M 1500MHz Getnoo Linux
Kvasir x86 – Pentium 4 2800MHz Getnoo Hardened Linux
Bast PPC – G3 350MHz Getnoo Linux
GP2X ARM – 920T 200MHz Debian Linux

Results

Time in seconds to find all prime numbers between 1 and one million

Langauge Inferno Nika Kvasir Bast GP2X
C 1.19 0.79 0.45 2.83 35.1
ObjC 1.19 0.8   2.83 43.4
C++ 1.93 1.06 1.1 4.76 50.1
Java 3.59 1.63 2.14 40.3  
C# 3.69 1.87   10.5 140
Awk 32.1 27.1 30 199 2065
Perl 38.2 21 23.3 145 1280
PHP 15.1 8.89 13.4 64.9 758
Python 54 38 43.8 211 1526
Lisp 10.4 5.19   36.3 2674

Primes x86 PPC Arm Unscaled

As you can see, of course, the 200MHz ARM GP2X was destroyed by everyone else. This really doesn’t tell us anything at all actually.

Clearly scaling needed to be introduced so that we could see how the languages faired on the different architectures. All being equal I assumed that GCC would give the best optimization per platform and that the C results would be the least skewed, so I choose to represent all the results as multiples of the C time results, or divide all the times by the time that the C prime number finder took on that computer. It yielded the following results:

Time in multiples of C time to find all prime numbers between 1 and one million

Language Inferno Nika Kvasir Bast GP2X
C 1.00 1.00 1.00 1.00 1.00
ObjC 1.00 1.01   1.00 1.24
C++ 1.62 1.34 2.44 1.68 1.43
Java 3.02 2.06 4.76 14.24  
C# 3.10 2.37   3.71 3.99
Awk 26.97 34.30 66.67 70.32 58.83
Perl 32.10 26.58 51.78 51.24 36.47
PHP 12.69 11.25 29.78 22.93 21.60
Python 45.38 48.10 97.33 74.56 43.48
Lisp 8.74 6.57   12.83 76.18

Primes x86 PPC Arm Scaled

Analysis

x86 machines

You would think we could use these 3 machines to establish a baseline but we find a fairly large variance in results here. First of all, inexplicably, Inferno is the only machine to have a worse result for Perl than Awk. Also, we can see that when it comes to the interpreted languages, Kvasir is near the worst in all cases. I would attribute this to Hardened Gentoo adding very visable overhead to the interpreters. It seems to show there is a definite cost for added security in terms of efficiency.

PPC

The most notable result here is the Java result which is clearly markedly bad. Java for the PPC could use some optimization. C# care of mono on the other hand is competitive. The rest of the interpreted languages also lag behind on the PPC, although Lisp (SBCL) comes pretty close the the x86 scaled results.

ARM

And now what you’ve been waiting for, the ARM results for the GP2X. Well, Sadly, I just couldn’t really get Java for the GP2X. Sun Java appeared to be unavailable. I could have used something like Kaffe or Jikes, but it probably wouldn’t have been to fair, and also, on Debian, they insisted in pulling in some parts of Xorg as dependencies, and I didn’t have a lot of space to play around with on the GP2X. Kind of lame, oh well. Again, as with the PPC, C# care of mono is very respectable. Other than that, the Lisp result was way off by a staggering amount, but that can be explained by SBCL not being available for ARM. I had to resort to the clearly slower CLisp compiler/’interpreter.’

Languages

As much hype as there is around Python these days, it appears to be fairly slow across the board compared to other interpreted languages. PHP on fact is the fastest of PERL, PHP, and Python on all tested architectures, so if you need speed and yet still the flexibility of an interpreted language, you might seriously want to use the CLI version of PHP. It seems the interpreter is very optimized across the board.

If you happened to be on an architectures supported by a good Lisp compiler like SBCL it is also a very attractive and viable option, but unfortunately for ARM, it looks like CLisp isn’t, speed wise anyway.

As for Java, hopefully the GPLing of it will allow it to a) be further optimized for alternate architectures like PPC, and b) for it to be fully ported to ‘new’ architectures like ARM. If you’re looking for speed but the middle ground flexibility of a VM language than in the mean time C# in the form of Mono is a fantastic looking choice.

Note: Ruby has again been omitted because until 2.0 is released, which includes a real VM, it is sadly not even remotely competitive in this area (math computation). It is about an order of magnitude slower than any of the other interpreted languages. Ruby 1.9 CVS last I checked (half a year ago) was competitive with Python :)

And that’s it from me on the Primes front (at east until I acquire a Sparc box :P …)