FYI Kotlin programs on benchmarks game

Yeah. Benchmark game was always too opionated. It does not use JMH for JVM-based tests and the choise of problems gives a lot of advantage to lower-level languages like C. In this case the results a laughtable, because the performance of kotlin program should be exactly the same as for java program excluding some rare border cases with lambdas.

People probably will fix that in no time. But the project itself is not fair to JVM from the beginning.

2 Likes

It does not use JMH for JVM-based tests…

JMH timings have been made.

1 Like

Glad to hear about it. JMH could actually demonstrate a minor slowdown because there are some JVM optimizations that work differently in micro-benchmark environment.

But the project itself is not fair to JVM from the beginning.

On the contrary.

I’m seriously doubting the validity of some of these results in regards to the environment they were run on, take for example the binary-trees benchmark:

| source | secs  | mem     | gz  | busy  | cpu load        |
|--------|-------|---------|-----|-------|-----------------|
| Java   | 8.32  | 953,620 | 835 | 28.37 | 95% 87% 81% 77% | 
| Kotlin | 20.97 | 6,656   | 769 | 64.29 | 78% 81% 75% 72% |

Java source
kotlin source

The Kotlin version is a near 1 to 1 translation of the Java version, but is 2.5 times slower and uses 99% less memory?

Edit: In fact this seems true for all benchmarks, they all seem to be using about the same amount of memory no matter the test. Is there some kind of aggressive GC going on?

2 Likes

Is there some kind of aggressive GC going on?

Good question.
The build and run options for each program are as-shown in the program log.

However, the as-shipped kotlin-compiler-1.3.61/bin/kotlinc file includes this –
[ -n "$JAVA_OPTS" ] || JAVA_OPTS="-Xmx256M -Xms32M"

I suppose we need to prevent that being default, and set the Java default instead.

Is there some kind of aggressive GC going on?

Now, the initial settings for the Kotlin programs are the same as the default settings for the Java programs -Xmx984M -Xms32M

same amount of memory no matter the test

That was still a problem, so I’ve changed the way the programs are run.

Now, instead-of –

kotlin -classpath fannkuchredux.jar fannkuchredux 12

– the command line is something like –

java -classpath ./fannkuchredux.jar fannkuchredux 12

That seems to give comparable memory-use measurements.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/java-kotlin.html

Great! The test results are starting to look a lot more like one would expect from a Java -> Kotlin translation.

On the topic of kotlin-like code, are external dependencies allowed at all? I’m sure some people would like to rewrite some of the parallel benchmarks with coroutines. While the stdlib supports them, the actual, official implementation is delegated to a dependency to reduce build size. A similar thing with Kotlin’s atomic data types.

…external dependencies allowed at all? … official implementation…

Should be OK. (Along with the program source, provide compile / run command-lines that don’t require maven.)

1 Like

The main optimization that allow languages to beat others on the benchmarckgame is about using SIMD.

If someone implement AVX~256 to a kotlin algorithm it should be multiple time faster.

Sadly kotlin nor Java have first class support for explicit SIMD.
But there are some libraries (might be useful https://github.com/JetBrains-Research/Viktor)

The real solution would be to use the Vector API
https://openjdk.java.net/jeps/338
Being the same API, one could contribute both for kotlin and Java at the same time :slight_smile:

I invite contributors to take inspiration from the c++ or rust algorithm implementations

BTW off-topic: OpenMP support for Kotlin/kotlin native would be really nice

What about graalVM and ZGC/Shenandoah impact on performance on those benchmarcks?

From May 2019 —

https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/java-substratevm.html

Straightforward “idiomatic” Kotlin programs that help show why someone might prefer Kotlin over Java would be wonderful !

Hopefully there’s someone with enough enthusiasm to contribute Kotlin programs that improve on those currently shown.

He probably meant JIT mode.It is much faster on some large math problems due to better escape analysis, but I do not think there will be difference for such small examples with primitives. Substrate currently is not meant for speed. Especially not in numeric tascs.

1 Like

It’s been a while since I looked, so —

https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/java-graalvm.html

Hopefully there’s someone with enough enthusiasm…

Not so far.

kotlinc-jvm 1.4-M1 (JRE 14+36-1461)

looked into the site, there are two java-programs for pi and the first one is compereable, but when you’re cchecking, it checks against the fastest (the spiedier) ont.

source secs mem gz busy cpu load
Java-2 3.07 38.408 938 3.39 0% 8% 99% 3%
Java-1 19.52 201,292 800 21.13 2% 57% 45% 4%
Kotlin 20.97 6,656 769 64.29 78% 81% 75% 72%

Although it seems that kotlin somehow traded memory for cpu-intensivity…

there are two java-programs for pi

Hopefully there’s someone with enough enthusiasm to contribute a Kotlin program that uses Gmp (like the faster of those Java programs).

This is a lot like extortion.

It’s a lot like crowdsourcing.

What contributors gain from that seems to have a lot to do with their own curiousity —

I will say this, the overall amount of time I’ve spent on writing the Julia benchmarks is far higher than it would have been for someone already familiar with writing high performance code. It’s been a great learning experience for me.