Why Kotlin Native?

Language Design
21

I would be really surprised if they’d come out with a Kotlin native compiler that doesn’t also compile Java. That would just split the ecosystem for no reason, especially as Java is basically a subset of Kotlin in semantics. I wouldn’t be surprised if it will just take class files as input. Andrey already said they were going to implement a GC.

My interpretation of what Jetbrains people have said about Kotlin Native is that they think IoT/embedded is going to grow a lot in the future, which is probably true. These devices are getting plentiful and powerful enough that bare metal languages (C/C++/D/Rust) aren’t a necessity anymore and higher level languages make sense, but not powerful enough to run a full JVM. There aren’t many well known languages in this space of high level + native compiled. What I can think of there’d be Go, Java with existing AOT compilers, or native compiled .Net, and Go is still a much smaller ecosystem than the other two. I think Jetbrains thinks they can create a good enough alternative with Kotlin Native to have an impact. I’m not very familiar with the state of existing Java AOT compilers, but I wouldn’t be too surprised if Jetbrains thinks that existing tools are not good enough and it will be easier to create a new compiler based on LLVM. And a Java-ecosystem based language would be uniquely suited for that since it is the only major ecosystem that is mostly self contained, in the sense that it is very rare for Java apps and libraries to have mandatory dependencies on native libraries. Which is unlike the .Net, Go, and any other open source language ecosystem.

A Kotlin Native offering would offer the same selling points as Kotlin/JVM but for the IoT and embedded market. I.e. a language that is very easy to learn for lots of developers and a familiar environment with a huge library ecosystem. Splitting the Kotlin ecosystem by not compiling Java libs would be a bit nonsensical, since then you would just have a new language and a new ecosystem with few libraries, which is not a very attractive proposition to a prospecting IoT developer.

The only thing I’m missing is how exactly Jetbrains plans to profit from this. Selling more tools to IoT devs is nice, but maybe they’re also planning to be an IoT or Kotlin consulting company or something like that.

2 Likes
22

I agree with @JanKanis, and disagree with my teammates from JetBrains that we need to compile Kotlin directly to LLVM. Unfortunately, I could not push my opinion to my teammates, so it remains my personal opinion.

Kotlin is an awesome languages for JVM, created WRT good interop with Java and with tons of Java libraries. Indeed, there are some disadvantages in JVM, at least, in OpenJDK, like large runtime size, JIT (as opposed to AOT) which can be inacceptable in some cases, unpredictable GC pauses, etc. That does not mean, however, that we should compile Kotlin ourselves directly to LLVM. This means that we need more appropriate JVM implementation. There are some, like RoboVM, MOE, etc. We can pick one and contribute our code there. We can even invent our own JVM AOT, it’s not that hard, at least comparing to Kotlin Native which has to implement semantics which is close to JVM. I see some advantages there:

  • We don’t have to do large amout of work, we can easily start with what we have and then make things better.
  • We could get a good adoption from people who have experience with Java and a little unsure to start using Kotlin. With JVM AOT approach they could write most of their code in Java and start to write some of the code in Kotlin (and fall in love with it). With current approach these people either refuse to port their app to iOS at all or discover and use MOE.

Let me tell about some myths about “ugly monstrous JVM”

  1. JDK is not necessary large. It is, but we can cut only parts necessary for embedded/AOT/iOS/games/whatever else, and it can be small enough. We can even use minifier like proguard or write our own one, which is capable of eliminating unused classes and methods.
  2. JVM is not necessarily JIT. We can sacrifice some ClassLoader features to have chance of creating pure AOT. AFAIK, RoboVM does not support class loaders, and it’s not a big limitation for iOS and embedded development.
  3. JVM implementation is not tied to GC, any automatic memory management mechanism is acceptable, including ARC, region-based memory management, etc.
  4. JVM is heavyweight. JVM bytecode consists of very simple instructions that remind of assembler. Full spec describes about 200 instruction versus x86 architecture which has thousands of instructions. Most of instructions are like iadd, fadd, isub, fsub, etc, which are trivially mapped to LLVM IR.
  5. GC does not necessarily produces unpredictable pauses, there are GC scheduling algorithms that give strong guarantees of upper limit of time spent in GC.
5 Likes
23

So @Alexey.Andreev you’re saying Kotlin Native will only accept Kotlin code as input and not Java? I’ll be sitting here being very surprised then. And what’s the story for the standard class libraries and e.g. Android framework code?

24

This will require licensing from Oracle and formal confirmation of full Java SE compatibility test suite. Otherwise they will sue JetBrains to NPE. They skipped RoboVM and other minor implementations due to insignificance and short life time.

Tell this to guys from excelsior. With stable implementation for x86, x64 and arm took like 4 years.

This is directly forbidden by licensing. This kind of cuts are formal reason for Oracle - Google ongoing trial.

2 Likes
25

This will require licensing from Oracle and formal confirmation of full Java SE compatibility test suite. Otherwise they will sue JetBrains to NPE. They skipped RoboVM and other minor implementations due to insignificance and short life time.

I’m not sure why this require licensing. Calling this implementation “Java SE” implementation will require passing JCK. Can Oracle sue product not tagged as “Java SE”? If can, is this so hard to pass JCK?

Tell this to guys from excelsior. With stable implementation for x86, x64 and arm took like 4 years.

I’m not sure how they actually implemented their AOT compiler. Did they write codegen themselves or they used LLVM? In the first case they had to create a good codegen, and creating a good efficient codegen is very complicated task. Also, they support class loading, therefore they have JIT as well. Creating subset of JDK (say, without ability to create new class loaders) won’t require JIT, which reduces amout of work. Also, did they implement JDK classes themself? If they used forked OpenJDK and licensed their product under LGPL, they also could reduce amount of work. Also, they support threads and memory model, which is quite complicated, even with LLVM. I believe, it’s possible to create something like green threads upon runtime coroutines.

This is directly forbidden by licensing. This kind of cuts are formal reason for Oracle - Google ongoing trial.

Can you provide a proof link? If this is the case, can they sue for automatic tool that takes only part of JDK and compiles it to native code? I guess it’s all about distribution, I don’t say about distributing cut version of JDK.

26

There is another issue. If you want to pass the JCK you will effectively have to build a JVM. The JVM is a complex beast and the Java standard library is tightly linked with it. Looking at it, the biggest paintpoints would probably be classloaders (the need to be able to load class files on demand), reflection (but Kotlin has its own reflection API anyway) and JNI.

A big win for native would be reasonably straightforward interop with existing native libraries, which means it needs to have a way to interact with C-style API’s and perhaps even C++ (perhaps with some limitations), but at least non-gc languages.

If you want to use the full power of the java standard library, why not stick with the JVM? Long term I see native as both native kotlin replacements and native exising library interop. Doing that would allow the advantages of being able to use the same code base for various platforms without having the tight semantics (and the hefty cost) of trying to stay very close to the JVM. It would also be in line with the JS target that does not stick too close to JVM semantics either.

1 Like
27

Anybody can sue for anything. Thats how courts work. If Kotlin becomes big enough that is just inevitable. The main difference is Oracle being very aggressive with Java copyrights and patents lately. Compare this say to Microsoft with Mono which they basically just bought. Making Java compat shims is like touching lava right now. It definitely requires serious legal analysis.

It is GPL with linking exception, not LGPL. It is very viral license and will require some tricky solutions to stay on apache license if you want to include compilation results into single binary. For example AOT compilation with static linking with GPLed OpenJDK may transfer GPL to compiled end user code. Maybe. Depends. I never read OpenJDK linking exception clause.

Google dodged bullet from Oracle by claiming fair use of copyrighted API. This means software API’s are copyrightable as seen from USA courts. Current appeal by Oracle hopefully will change this.
More details:
Google LLC v. Oracle America, Inc. - Wikipedia.

28

I agree with @pdvrieze that Kotlin will need to interop with whatever libraries/frameworks are present on the platform that Kotlin Native is targeting. Trying to take the JVM ecosystem to every platform isn’t feasible since the ecosystem itself is tightly coupled to the JVM. There might be even be some serious legal issues that could arise from the Oracle vs Google case that is currently ongoing.

Take the Web platform for example where there are many useful libraries/frameworks that Kotlin developers WILL want to take advantage of. Any programming language that wants to target the Web (front-end, and possibly back-end) needs to seamlessly interop with the existing libraries/frameworks there. When it comes to memory management @Alexey.Andreev might have a point with Kotlin using something like ARC (Swift uses this system), instead of a GC.

Is ARC really a form of automatic memory management?

2 Likes
29

Yes, ARC is technically a form of automatic memory management but suffers from the drawback that it is unable to deal automatically with cyclic references. You therefore have to deal with these manually using weak references which is not always straightforward.

However, I believe JetBrains have already said that they’re building their own ‘native friendly’ GC for Kotlin Native. I don’t know whether this means that you’ll be able to suspend the GC when executing a critical piece of code, optionally use manual memory management or both. We’ll just have to wait and see.

30

Looking at this outdated document there are some other things that might be covered by Kotlin Native. See that native interop is being considered. Very interesting that Kotlin Native could have its own concurrency layer.

1 Like
31

I think Kotin Native makes sense. Golang has been a hit because of channels, coroutines and it allows you to produce a single executable file that’s easy to deploy. Kotlin will have all of that plus decent syntax.

If enough of the basic JVM libraries can be rewritten or replaced with Kotlin libraries, Kotlin may get an added boost.

Jetbrains has managed to produce a very nice language with relatively little investment. If native works as well as JVM, then it could be great.

1 Like
32

Wow! Amazing!
I expect Kotlin Native to beat Rust which is really a headache.
Is there somewhere I can track the status of this project?

3 Likes
33

Presumably the drive behind Kotlin native is to move the JetBrains IDEs off of the JVM.

34

Rust is designed as a much lower level language with e.g. semi-manual resource management. Kotlin (unless Kotlin Native is going to be very different from Kotlin-JVM in this regard) is a much higher level language. If you don’t need the speed & control Rust gives you something like Kotlin will likely be more suited to a task.

There’s no official release or announcement of Kotlin Native yet, we’re still waiting for that.

35

Unlikely. One of their reasons for creating Kotlin was to have something interoperable with Java because rewriting all of their code was not an option. And what advantage would a native IDE give over a JVM based one?

1 Like
36

When Kotlin Native eventually emerges, I wouldn’t be surprised to see plug-ins for AppCode (for iOS development) and CLion for everything else. These might be a more natural home for it than IntelliJ IDEA and would help to popularize the aforementioned IDEs.

I tried Rust myself but soon gave up. I found that my thinking was dominated by memory management and consequently it was difficult to concentrate on what I was actually trying to program!

1 Like
37

No, it’s not.

1 Like
38

Hi,

I am the CTO of Migeran and the Project Lead of the Multi-OS Engine (MOE) project. At Migeran, we developed Migeran for iOS, the original product that was acquired by Intel and its technology used in MOE.

TL;DR: I see a lot of opportunities for cooperation with Kotlin Native. Feel free to contact me directly, if you are interested as well.

I assume, that Kotlin Native (while including special support for Kotlin specific features), will be a generic AOT compilation based VM capable running any AOT compatible JVM app. I don’t see the use case for a VM that can only run code written in Kotlin directly.

First, a few notes about MOE:

For MOE 2.0 we developed a completely new LLVM backend for ART. It is based on Android N. For those, who remember the early days of ART, this new backend has no relation to the original “portable” backend. It is essentially a new backend for ART’s optimizing compiler. Our backend is already functional, we are working on cleaning the code before releasing the first alpha.

MOE of course runs Kotlin apps just fine (including any language that would run on Android ART).

I am not opposed to the idea to add new, Kotlin specific runtime features to MOE. The benefit would be, that you don’t have to start everything from scratch. The tradeoff is, that a lot of design decisions are already made in ART, although a lot of those assumptions can be changed (and we already changed some of them in MOE 2.0).

I can also understand why you (Kotlin Native developers) would think, that starting from scratch will yield better results for your use cases. VM design is a complex topic, with lots of different design options along the way. A nice example of this is comparing MOE with RoboVM. The high level view is that both projects solve the same problem: a working JVM for iOS. But when you look closer, in almost every design decision the projects went in a different direction. Just a few examples:

  • RoboVM started from scratch, we started from the Android VM (first Dalvik, then moved to ART when it was first released by Google)
  • RoboVM created a VM specific ObjC binding framework (Bro), we created Nat/J on top of JNI, so it works with any Java VM. (Side note: This also means, that if Kotlin Native will be JNI compatible, it can reuse MOE’s iOS bindings.)
  • RoboVM is full AOT, MOE (ART) includes an interpreter. It is used during class initialisation and debugging: methods containing breakpoints or methods that are being stepped through execute on the interpreter. (This also means that MOE can theoretically load new classes (in dex format) at runtime. These will run on the interpreter, we don’t recommend or support it, but should work. A valid use case for this could be a development framework where code can be created and pushed into a running Playground type app on the fly. Think Kotlin REPL running on an iOS device or IoT)

I see multiple areas where we could cooperate. These areas are pretty much independent from each other.

  1. Turning ART into a true Kotlin Native environment

    If you think that you could live with re-using ART, it could be extended / refactored into a very nice generic VM platform, where its components are runtime exchangeable through a plugin API. This would make it possible to load Kotlin specific VM extensions at runtime, and (if Google plays along), this could also benefit any Kotlin app running on Android.

    For other platforms, MOE’s LLVM backend can be used to provide superb performance, and it can be optimized to handle Kotlin workloads even better by using the Kotlin specific extensions in the VM directly.

    I saw in a Kotlin Native document, that you are considering optional support for ARC / reference counting based memory management. I would also very much like to see support for this in a JVM, so ideally the plugin interface of ART would make the implementation of different memory management strategies possible. Classes could opt-in for a specific memory management model using the right ClassLoader.

  2. Unified Java Runtime library

    The biggest issue in the JVM world is that the runtime library versions are very fragmented. There are 2 big players (as you all know): Android and Oracle Java.

    Android 7 propelled the platform compatibility level from the distant, obsolete past of (almost) Java 7 to the not so distant and soon to be obsolete past with adding limited Java 8 support. Their approach with the Jack toolchain is currently being reconsidered (I am being diplomatic here :slight_smile: ). They realized, that reimplementing the Java 8 API is technically, financially and probably legally is just not feasible. So they did the right thing, and moved to use OpenJDK as the basis. But they did it in a way that effectively forks OpenJDK in the ugliest possible manner. I won’t go into detail here, check out the ojluni module in AOSP if you dare. :slight_smile:

    Oracle has its own problems. The release was pushed back multiple times already (especially the module framework seems to be hard to get right - which is understandable). The world is now mostly using Java 8, but there are still lots of people stuck in the dark ages of Java 6 or 7 due to legacy environments.

    This level of fragmentation is hurting the Java / JVM ecosystem. My proposal is, that ART (and any other VM) should move to the Java / OpenJDK 9 library as soon as possible. Changes on top of the Java 9 runtime library should be kept as a minimum, and each project should track Java 9 development head as closely as possible.

    I also propose, that a new base module of the runtime library is defined for AOT VMs. Note, that Oracle is also working in this space with a VM called the GraalVM and with AOT support for HotSpot, so they might be willing to accept these changes into the mainline. Even if Oracle does not accept these changes, the different VMs (including stock Android, if Google plays along) should be able to share their runtime library repositories in a common project and reduce the synchronization and maintenance burden on each project.

  3. Kotlin iOS bindings

    As I already mentioned, MOE has a fairly complete, automatically generated binding for iOS. It is compatible with any JNI compatible VM. The generator is using the Objective-C headers, but it does not yet use the extra attributes that were added for Swift. It is following the original ObjC APIs very closely (e.g. uses alloc().init*() based 2 phase construction).

    It is on our roadmap to move to a more modern binding, modelled after the Swift bindings. We are considering to move to Kotlin for the next version of the API bindings. There are features in ObjC / Swift, that cannot be easily / nicely reflected in Java (e.g. extension classes), Kotlin would make this easier. This would also mean that Kotlin Native would have access to high quality, Kotlin native bindings for iOS, tvOS and (potentially) WatchOS.

    This could be also an area, where we could share our efforts.

To summarize: I see a lot of opportunities for cooperation, and I would be happy to discuss these topics further.
Feel free to reach out to me here, or contact me directly.

Best Regards,
Gergely

5 Likes
39

Thanks for reaching out, and sorry for the late reply!

In fact, we are not trying to do a generic AOT for JVM. We want Kotlin code to be portable between platforms, but we do not to make all Kotlin code portable. Please check out this blog post for more details: https://blog.jetbrains.com/kotlin/2017/04/kotlinnative-tech-preview-kotlin-without-a-vm/

I think we could potentially find a common ground when it comes to iOS bindings, at least we should investigate this.

4 Likes
40

I just want to say that it would be cool to have Konan - Kotlin Native in clang (llvm) to use it everywhere.