Some facts about programming languages performance

Introduction

I've read a lot about the performance of different programming languages. In those discussions the results of benchmarks such as The Computer Language Shootout Benchmarks are often cited.

As a graduate student running long numerical simulations I have two main concerns about my codes:

1. They must run as fast as possible
2. They must be easy to modify and test. After all a fast but wrong answer is not useful at all.

Both objectives seem orthogonal, because the fastest languages are in general associated with the most difficult to debug and modify codes. That is the main reason why I have spent so much time searching for the "perfect language": the one that is fast, reliable, and easy to use. In that process I have used several languages to write my codes or part of them. I looked into C/C++, Fortran95/2003, D, Java, and a combination of Python/Fortran95 and Python/C++.

By experience I learned that the performance of codes is more related to the compiler or virtual machine (VM) and available libraries, than to the language itself. That fact is often neglected by comparisons like the The Computer Language Shootout Benchmarks where only one compiler/VM is used. So in general the conclusion of those test is that one language is faster/slower than another one for a given problem. To demonstrate this point I decided to run some of the benchmarks in the The Computer Language Shootout Benchmarks using different compilers for the same language. Because of limited time I decided to run few of them, so I picked up the ones more representative of numerical applications.

Results

All tests were run in a Pentium4 3GHz with 1.5 Gb of RAM running Linux OpenSuse 10.2 (kernel 2.6.18) . The figures below show normalized time, i.e. CPU time divided by the minimum time required to perform a specific test. Those figures show results for a single realization, but they are consistent with results from multiple realizations. I considered the following programming languages: Fortran 95, C, C++, D, and Java (server). To compile the tests I used some common optimization flags (e.g. -fast, -mtune=pentium4, etc).

The first figure shows the results for a classical nbody problem. We observe that all language/compiler combinations required an execution times less than 80% longer than the shortest time.It is interesting to see that runs using the same language (Fortran) but different compiler exhibits differences of ~40%. We observe that Sun Java 6 VM does a good job compiling the code and it is slightly faster than compiled code produced by Digital Mars D compiler.


The second figure shows the results for the spectral norm test. In this case most of the language/compiler pairs, with the exception of the GNU C and Fortran compiler and one of the Java VM, required a time that was less than 20% longer than the fastest code. Again we observe that Sun Java is slightly faster than some of the compiled codes (both D compilers).



The last figure shows the results for the partial sums benchmark. This test intensively uses primitives mathematical functions such as sin, cos, pow, etc. Therefore, it mostly measures the library implementation of those functions. We observe that results are separated in three groups. The fastest code is generated by Intel Fortran and C compilers. We believe those compilers are using very efficient math libraries to evaluate the primitive functions. All the other compiled codes (GNU Fortran and C, Sun Fortran, and D DMD and GDC) require between 3 to 4 times longer to run the same simulations. Finally, both Java VM required 13 and almost 16 times longer to compute the same results. This can be explained by the use of slow but more accurate primitive functions as explained here.



Conclusions

From this simple test we can conclude that

1. Variation in the performance of a code written in the same language but compiled or run with different compilers or VM can be much larger than the difference of similar codes written in different languages.
2. The results of these tests do not show a clear trend in the performance of a particular language. However, we observe that Fortan and C/C++ code is, in general, faster than other languages. That can be explained by the availability of good optimizer compilers and efficient libraries for those languages.
3. In some problems (e.g., partial sums test) the use of efficient libraries can have a much more significant performance impact that the choice of a programming language.
4. The results of these tests do not provide evidence to claim that traditional compiled languages are intrinsically faster than Just-in-Time (JIT) compiled languages (Java).
5. A programmer concerned about performance should use other criteria, e.g. clarity, expressiveness, available libraries, etc.; to choose a programming language without necessarily losing performance.