Some time ago I did simple comparison between some languages (most from JVM) in performance aspect.
Each application do same work:
- Reads text files with cartographic data from specified directory.
- Parses lines and extract information about segments with begin and end points.
- Calculates physical distance between segment and special point and if distance is less than the specified radius stores segment into map array.
Thus the following subsystems of language take part in competition:
- I/O (read big text files line-by-line);
- String parsing;
- Calculations with Double numbers;
- Working with Maps and Lists.
Source code of applications: LangBenchmark
Competitors:
- Orcale Java 8 (update 152)
- Scala 2.12
- Ceylon 1.3.3
- Kotlin 1.2.30
- Fantom 1.0.69
- Qt/C++ 5.10
- D 2.078.2
- C# 2015
Results for Windows 7 SP1 x64:
Small files:
- Java: 621ms, 1,00
- Kotlin: 667ms, 1,07
- Scala: 745ms, 1,20
- D: 797ms, 1,28
- C#: 1143ms, 1,84
- Ceylon(JavaLibs): 1160ms, 1,87
- Fantom: 1362ms, 2,19
- Qt/C++: 1378ms, 2,22
- Ceylon: 1479ms, 2,38
Big files:
- Java: 22932ms, 1,00
- Scala: 23013ms, 1,00
- Kotlin: 23300ms, 1,02
- Fantom: 32047ms, 1,40
- C#: 33349ms, 1,45
- Ceylon(JavaLibs): 38466ms, 1,68
- Qt/C++: 40444ms, 1,76
- Ceylon: failed - freezes.
- D: failed - “core.exception.OutOfMemoryError@src\core\exception.d(702): Memory allocation failed”
Results for Debian Stretch Stable:
Small files:
- Java: 612ms, 1,00
- Kotlin: 652ms, 1,07
- Scala: 686ms, 1,12
- D: 785ms, 1,28
- Qt/C++: 1023ms, 1,67
- Ceylon(JavaLibs): 1190ms, 1,94
- Fantom: 1356ms, 2,22
- Ceylon: 1480ms, 2,42
- C# (mono): 2119ms, 3,46
Big files:
- Java: 22161ms, 1,00
- Scala: 22625ms, 1,02
- Kotlin: 22865ms, 1,03
- D: 22876ms, 1,03
- Qt/C++: 31349ms, 1,41
- Ceylon(JavaLibs): 34664ms, 1,56
- Fantom: 40903ms, 1,85
- C#: 68038ms, 3,07
- Ceylon: failed - «Exception in thread «main» java.lang.OutOfMemoryError: GC overhead limit exceeded»
Kotlin has shown good result in performance on JVM - very close to Java.
It makes no sense to compare kotlin to java since in most cases you just use java classes to do actual operations. The same basically goes foe scala if it does not use something special.
You are not using JMH, so benchmarks are not very relevant any way. Without it you also can’t compare performance of different VMs.
Even though in applications on Kotlin, Scala and Java used same classes from JVM, but byte code of applications will be different. And this difference will affect performance.
I look at benchmark applications as on “blackbox” with:
- same source data: input files;
- same environment: hardware, OS and JVM (if applicable);
- same result: number of found segments;
- different time of execution - performance effect, ie result of benchmark.
From this point of view it’s no matter what inside “blackbox” - native application or JVM based or something else VMs.
I didn’t use JMH, but I have created launcher for my competitors which runs each application ten times and measures time - final result is average time for each application.
I am not sure about Ceylon, but Kotlin and Sclala do not generate different bytecode for java classes. It is the same. The only difference is in launcher code and class loading (the later generates the visible difference in time). JMH is not only about VM warm-up. I am not specialist, but you should read its description and some discussions about it. I was told that tests without jmh are not considered to be valid.
I’m talked about bytecode of application. Scala, Java, and Kotlin, naturally, have different compilers and generates different bytecodes.
JMH is not applicable in my case because I compare not only languages from JVM.
C# - is an application from another VM.
D - native application with own runtime.
Qt/C++ - is native application as well.
Fantom - has own environment over JVM.
And anyhow JMH is just framework written by peoples 
it doesn’t contain any magic inside.
I’m talked about bytecode of application. Scala, Java, and Kotlin, naturally, have different compilers and generates different bytecodes.
Not quite correct. They do generate bytecode for the caller part, but they do not generate bytecode for java library functions and objects like BufferedReader you use. I do not know Ceylon, but I think you’ve done something wrong to get such bad performance. Read operations should take exactly the same amount of time.
Yes, we on the same page. I also talked about the caller side.
My strategy was: use I/O library of language if it presents and uses JDK classes if not.
e.g. Ceylon has own implementation of I/O library and I use it, but it crashes on big files and I had to write a separate application on Ceylon which uses Java I/O. It marked as Ceylon (JavaLibs) in test and it works faster.
This is unlikely because I had a discussion with Ceylon authors about bad performance of Ceylon I/O. They checked my Ceylon implementation and didn’t have any comments about any bugs in code.
Ideally yes. But different languages have different implementations of I/O with different optimizations.