Some facts about programming languages performance

Introduction

I've read a lot about the performance of different programming languages. In those discussions the results of benchmarks such as The Computer Language Shootout Benchmarks are often cited.

As a graduate student running long numerical simulations I have two main concerns about my codes:

1. They must run as fast as possible
2. They must be easy to modify and test. After all a fast but wrong answer is not useful at all.

Both objectives seem orthogonal, because the fastest languages are in general associated with the most difficult to debug and modify codes. That is the main reason why I have spent so much time searching for the "perfect language": the one that is fast, reliable, and easy to use. In that process I have used several languages to write my codes or part of them. I looked into C/C++, Fortran95/2003, D, Java, and a combination of Python/Fortran95 and Python/C++.

By experience I learned that the performance of codes is more related to the compiler or virtual machine (VM) and available libraries, than to the language itself. That fact is often neglected by comparisons like the The Computer Language Shootout Benchmarks where only one compiler/VM is used. So in general the conclusion of those test is that one language is faster/slower than another one for a given problem. To demonstrate this point I decided to run some of the benchmarks in the The Computer Language Shootout Benchmarks using different compilers for the same language. Because of limited time I decided to run few of them, so I picked up the ones more representative of numerical applications.

Results

All tests were run in a Pentium4 3GHz with 1.5 Gb of RAM running Linux OpenSuse 10.2 (kernel 2.6.18) . The figures below show normalized time, i.e. CPU time divided by the minimum time required to perform a specific test. Those figures show results for a single realization, but they are consistent with results from multiple realizations. I considered the following programming languages: Fortran 95, C, C++, D, and Java (server). To compile the tests I used some common optimization flags (e.g. -fast, -mtune=pentium4, etc).

The first figure shows the results for a classical nbody problem. We observe that all language/compiler combinations required an execution times less than 80% longer than the shortest time.It is interesting to see that runs using the same language (Fortran) but different compiler exhibits differences of ~40%. We observe that Sun Java 6 VM does a good job compiling the code and it is slightly faster than compiled code produced by Digital Mars D compiler.


The second figure shows the results for the spectral norm test. In this case most of the language/compiler pairs, with the exception of the GNU C and Fortran compiler and one of the Java VM, required a time that was less than 20% longer than the fastest code. Again we observe that Sun Java is slightly faster than some of the compiled codes (both D compilers).



The last figure shows the results for the partial sums benchmark. This test intensively uses primitives mathematical functions such as sin, cos, pow, etc. Therefore, it mostly measures the library implementation of those functions. We observe that results are separated in three groups. The fastest code is generated by Intel Fortran and C compilers. We believe those compilers are using very efficient math libraries to evaluate the primitive functions. All the other compiled codes (GNU Fortran and C, Sun Fortran, and D DMD and GDC) require between 3 to 4 times longer to run the same simulations. Finally, both Java VM required 13 and almost 16 times longer to compute the same results. This can be explained by the use of slow but more accurate primitive functions as explained here.



Conclusions

From this simple test we can conclude that

1. Variation in the performance of a code written in the same language but compiled or run with different compilers or VM can be much larger than the difference of similar codes written in different languages.
2. The results of these tests do not show a clear trend in the performance of a particular language. However, we observe that Fortan and C/C++ code is, in general, faster than other languages. That can be explained by the availability of good optimizer compilers and efficient libraries for those languages.
3. In some problems (e.g., partial sums test) the use of efficient libraries can have a much more significant performance impact that the choice of a programming language.
4. The results of these tests do not provide evidence to claim that traditional compiled languages are intrinsically faster than Just-in-Time (JIT) compiled languages (Java).
5. A programmer concerned about performance should use other criteria, e.g. clarity, expressiveness, available libraries, etc.; to choose a programming language without necessarily losing performance.
   

Operator overloading in Java/Jython

Yesterday, I discovered that it is possible to add operator overloading to Java through Jython.

I created a small Point.java class:

import java.lang.String;
public class Point {
private double x, y;

public Point(double x, double y){
this.x = x; this.y = y;
}

public Point(Point other){
this.x = other.x; this.y = other.y;
}

public Point __add__(Point b) {
return new Point(this.x + b.x, this.y + b.y);
}

public String toString(){
String str = "("  + this.x  + "," + this.y + ")";
return str;
}

public String __repr__(){
String str = "Point: ("  + this.x  + "," + this.y + ")";
return str;
}
}

I compiled this class with: javac Point.java. In the same folder I created a testPoint.py file with:

import Point

a = Point(1.0,1.0)
b = Point(2.0,2.5)
print "a = ", a
print "b = ", b
c = a + b
print "a + b = ", c

Running: jython testPoint.py, produces

a = (1.0,1.0)
b = (2.0,2.5)
a + b = (3.0,3.5)

Language performance

Since long time I wanted to answer a simple question: what is the difference in performance of different programming languages when solving numerical problems? I've read a lot about the subject on the web. Unfortunately, I found many different opinions.

Some people say Java is too slow for number crunching, others say it can be faster than plain C for some problems. Finally, I found some posts that claim that C# in .Net is faster than Java. Since I'm thinking to rewrite part of one of my codes I thought it would be a good idea to answer this question before starting my new project.

As first step, I decided to use a single benchmark focused on floating point operations. In Java, Scimark2 is the standard. There are also version for the other two languages (C and C#) that I used in my comparison.

After using Linux as my main desktop for the last 4 years, I moved back to Windows XP some weeks ago. That's why I also wanted to check the effect of the OS in the final performance. Finally, I wanted to check a wide range of C compilers, Java VMs, and .Net implementations (well, I only know two...).

TODO: Add results...
Scimark2 results in Windows XP Home SP2



Scimark2 results in Linux Fedora 5