Here is a collection of benchmarks that I did myself on various free VMs and links to benchmarks done by others.
The machine that ran the test had a Athlon 1600+ (1.4Ghz) and 768MB. It was lightly used and I don't claim to have done scientific benchmarking. All times were measured with /usr/bin/time -p and the best of three runs is given. (Note that some tests measure elapsed time themselves so the time -p values are for reference to see how much time the complete run actually took. Which is relevant in some cases.)
Since a picture tells more then a thousand words, first some pictures.
This is the interpretation of the results that I send to the GNU Classpath mailinglist plus some comments made by others. Please see the original mailinglist thread for more comments and discussion.
Looking at the results there are clearly three groups: the plain interpreters (kissme, gij and sablevm), the jitters (kaffe and ikvm) and the ahead of time compiler (gcj). The jitters are clearly much faster then the interpreters, and the ahead of time compiler is clearly much faster then the jitters.
Interesting is that ikvm has results comparable with kaffe, but often takes more time "preparing" the classes. This can be clearly seen with the Linpack benchmark where the ikvm score and kaffe score are almost identical, but when looking at the real time spend you see that ikvm consumes much more processor cycles.
Jeroen Frijters notes that ikvm also has a precompiler that can be used as followes: ikvmc -out:scimark.exe -target:exe -reference:classpath.dll -main:jnt.scimark2.commandline scimark2lib.jar. When you run the exe, make sure it is in the same directory as the ikvm dlls. Also, if you give an incorrect main class name, ikvmc gives no error but the resulting exe doesn't have an entry point (i.e. it is a dll).
Paolo Molaro notes that mono has a -O=all option. It's necessary to turn on all the optimizations and it may have a significant effect on some benchmarks (though it also increases JIT time and there may be some regressions, too, that needs to be investigated).
A hard disk failure made me unable to try out the above two optimizations.
Since VMs have different goals, such as small disk or memory footprint, portability, efficiency, etc. the results may not be that interesting since they are very limited to speed issues. As Etienne Gagnon notes: SableVM's goal is *NOT* to beat other VMs in micro/mini/macro-benchmark contests, but to provide a highly versatile and portable, yet *efficient* framework.
Since gcj compiled code seems to be very fast but the gij interpreter is not very fast it might be interesting to combine gcj with one of the jitters or make one of the jitters be able to use precompiled binaries produced by gcj (for example to precompile all core classes used by the VM).
Jeroen Frijter notes: At least for IKVM (and I suspect for others as well), the floating point benchmarks are probably inflated since I don't think it implements FP correctly (i.e. it uses x86 FP and not the JVM model).
And in the end it just matters to have a choice in free powerful and efficient development environments to make the creation of Free Software for all users easier. But it seems that having a benchmark contest from time to time also helps that goal!
Here is the raw data that the above graphs and interpretation is based on.
Scimark2 Kissme: Composite Score: 3.0156149099413163E174 FFT (1024): 1.5078077289324666E175 SOR (100x100): -1.1107352312578372E61 Monte Carlo : 1.6082285944663933E129 Sparse matmult (N=1000, nz=5000): 2.6108808678857117E25 LU (100x100): 3.5030103117925084 real 26.99 user 26.82 sys 0.05 gij: Composite Score: 5.145989276632694 FFT (1024): 2.648052903699278 SOR (100x100): 8.881613884448855 Monte Carlo : 1.387464107626074 Sparse matmult (N=1000, nz=5000): 5.551639995055836 LU (100x100): 7.261175492333425 real 31.30 user 31.19 sys 0.10 SableVM: Composite Score: 7.329552111085732 FFT (1024): 4.404292204203279 SOR (100x100): 13.062330882926906 Monte Carlo : 2.369663222573336 Sparse matmult (N=1000, nz=5000): 7.329992764243664 LU (100x100): 9.481481481481481 real 28.23 user 26.95 sys 0.11 Kaffe: Composite Score: 60.46979156068268 FFT (1024): 30.891199299334463 SOR (100x100): 148.2273083076923 Monte Carlo : 12.628691263262526 Sparse matmult (N=1000, nz=5000): 54.10832237609153 LU (100x100): 56.49343655703252 real 31.17 user 31.10 sys 0.00 ikvm: Composite Score: 100.9979186697824 FFT (1024): 59.79980402491375 SOR (100x100): 224.2731618120265 Monte Carlo : 34.02223834505183 Sparse matmult (N=1000, nz=5000): 83.7413726183124 LU (100x100): 103.1530165486078 real 35.60 user 35.42 sys 0.09 gcj: Composite Score: 217.51089337171615 FFT (1024): 225.23768378150965 SOR (100x100): 305.57485467536026 Monte Carlo : 15.335663953302584 Sparse matmult (N=1000, nz=5000): 235.82583613959164 LU (100x100): 305.5804283088167 real 28.49 user 28.44 sys 0.06 Sieve Kissme: 634 iterations in 10.008 seconds Sieve score = 63 real 10.10 user 9.98 sys 0.02 SableVM: 2355 iterations in 10.0 seconds Sieve score = 236 real 10.08 user 10.08 sys 0.01 gij: 2692 iterations in 10.0 seconds Sieve score = 269 real 10.10 user 10.05 sys 0.05 ikvm: 40937 iterations in 10.0 seconds Sieve score = 4094 real 11.32 user 11.24 sys 0.05 Kaffe: 48611 iterations in 10.0 seconds Sieve score = 4861 real 10.11 user 10.10 sys 0.01 gcj: 91621 iterations in 10.0 seconds Sieve score = 9162 real 10.03 user 9.95 sys 0.06 Linpack Kissme: Mflops/s: 2.289 Time: 36.62s Norm Res: 0.0 Precision: 2.220446049250313E-16 real 40.73 user 40.62 sys 0.10 gij: Mflops/s: 6.078 Time: 13.79s Norm Res: 5.81 Precision: 2.220446049250313E-16 real 14.74 user 14.59 sys 0.14 SableVM: Mflops/s: 8.45 Time: 9.92s Norm Res: 5.81 Precision: 2.220446049250313E-16 real 10.88 user 10.86 sys 0.02 ikvm: Mflops/s: 60.312 Time: 1.39s Norm Res: 5.68 Precision: 2.220446049250313E-16.0 real 5.03 user 4.92 sys 0.06 Kaffe: Mflops/s: 61.282 Time: 1.37s Norm Res: 4.28 Precision: 2.220446049250313E-16 real 1.92 user 1.89 sys 0.03 gcj: Mflops/s: 79.163 Time: 1.06s Norm Res: 4.28 Precision: 2.220446049250313E-16 real 1.41 user 1.37 sys 0.03
