Compiling cross-module calls with invokedynamic

1

This paper proposes an interesting idea:

It compiles cross-module method calls with invokedynamic, supported by a bit of code in e.g. the standard library of the language, such that changes in a dependency that would be source compatible but not binary compatible are automatically handled.

Such a thing could be integrated into the Kotlin compiler transparently when it targets Java 8. It would mean developers don’t need to understand the difference between source and binary compatibility any more (a subtle topic), and could make incremental compilation more efficient. The reported overhead is nearly zero.

2

This is a very compelling idea, especially for Kotlin with its default parameters for functions, where adding yet another parameter, even with a default, is source-compatible but is not binary-compatible. This distinction does make it harder to maintain binary compatibility for library authors.

However, Kotlin is big on Android, which does not have invokedynamic in practice, so having implemented this feature we’ll actually make a bad favor for an ability to reuse Kotlin’s libraries on Android. The other way to look at it, is that making library authors’ life easier is not and never was a priority for Kotlin. Most of the users of the language are application developers who do not care about those binary/source compatibility issues at all.

1 Like
3

adding yet another parameter, even with a default, is source-compatible but is not binary-compatible

In the absence of @JvmOverloads you mean?

That’s pretty nasty. I hadn’t actually thought about the binary compatibility of adding default arguments before … I vaguely recall the way this is compiled is rather tricky in the absence of the overloads annotation, with some bitmasking being done for efficiency, right?

I like to think that I’m more familiar with Kotlin and bytecode than most developers - even so, I can imagine myself adding a default parameter and not realising I’d broken the library ABI.

At the very least it would be helpful for there to be a page in the documentation discussing what changes are and are not binary compatible.

Now to the wider question:

  1. In the wake of winning the Oracle lawsuits, it seems that Android has re-committed to upgrading its support for the Java platform. Android O introduces the java.lang.invoke package, with classes that are only useful with invokedynamic like CallSite. Whilst I don’t know what they plan to do, the fact that Java 9 is doubling down on indy-based compilation means they’ll eventually either have to add support for indy or make their “de-sugarer” even more complex. I suspect they will eventually implement this opcode.
  2. Yes, Kotlin is big on Android, but Kotlin can also get big in other areas: on the server and desktop. App containers don’t really exist on Android but they are common in the rest of the Java space, and “eliminate binary compatibility as a thing to think about” would be a really useful feature here. Why should Android hold back server/desktop Kotlin when a feature only affects compilation strategy?
  3. You are using Kotlin in IntelliJ. This is good. But IntelliJ is an app container and thus it’s in JetBrains’ own interest to ensure that Kotlin plugins that benefit from Kotlin APIs exposed by IntelliJ don’t needlessly break across IntelliJ upgrades. An indy-based compilation strategy could assist here.

Overall I think this would be a good upgrade. Eliminating the need to think about binary compatibility is the kind of mental simplification that Kotlin tends to support (e.g. hiding the differences between boxed and unboxed types as far as possible).

(btw to explain my keenness on this feature, I am developing an app container in Kotlin too)

4

My preference would be to start with documentation and additional tools that would help library authors to become aware about binary compatibility.

5

Yes, that’s certainly the low cost first step. Fixing an issue so it vanishes is preferable to documenting it though.

6

It may very well be possible to take a library and bytecode that uses it and have a tool determine all linkage failures ahead of time. It may be some work to do so, but I suspect not that much in many cases. Of course if you mess about with object hierarchy and polymorphism there are many ways it can break (but is duck-typing on object instances really a sensible implementation, even as “compatibility”)? At the same time some functions exist on multiple interfaces (Closeable.close vs Autocloseable.close) where using the “correct” one can make a difference in binary compatibility.

Of course none of any tool can fix changed static finals / constants that were inlined at call site by the compiler.

7

On a side note, adding @JvmOverloads does not fix the fact that, in Kotlin, an addition of a new parameter with default is not a binary-compatible change (even with @JvmOverloads it is not binary-compatible that is).

8

There’s a list of rules we prepared for the standard library to ensure its binary compatibility: https://github.com/JetBrains/kotlin/tree/master/libraries/tools/binary-compatibility-validator#what-makes-an-incompatible-change-to-the-public-binary-api

9

Thanks Ilya, but that document doesn’t list the “adding a default parameter value” rule. It seems to be written from the perspective of what changes a JVM-level ABI can make, but not how Kotlin-level source rules map to it. It’s the former that developers need to know (or not, if some sort of runtime adaptation is implemented).

10

By the way, is any pre-existing/non-indy solution planned for this default parameters breaking ABI compatibility issue? It seems this will hurt both stdlib developers and my own project, where we are preparing to commit to an ABI for the long term … Java style!

11

We had introduced Deprecated(HIDDEN) for that. When you need to add another optional parameter to a method, you keep the original method annotating it with Deprecated(DeprecationLevel.HIDDEN) — this way it isn’t accessible from the source code, but still keep compiled to the bytecode. Then you add a new one with the additional parameter, probably delegating the implementation of the old one to it.

12

How does that affect Java interop? It doesn’t understand Deprecated=HIDDEN, right? How does it work if you specify a default to the parameter too?

13

The rule of thumb is: adding a new method to an existing class does not break ABI, while every other change does break ABI. There are really few other exceptions, and they are not really worth remembering in practice. My second, most often used rule, is that you can move methods up the hierarchy and add new superclasses and superinterfaces without breaking ABI (but be careful to fully preserve the signature of methods).

1 Like
14

You can probably get away with that sort of mental shortcut in something like a standard library, where there are lots of small types and individual utility methods. It’s a bit harder for us with large types that model complex business concepts and services, where they may need to evolve over many years to incorporate lots of new features.

For example

  • adding a val/var is ABI compatible because it translates to a private field + get/set methods. Adding vals/vars is something we will probably need to do a lot.
  • What about adding a companion object? I’ll have to check how it’s compiled.
  • What about adding an inlined function? AFAIK it does show up as an added method in the bytecode?
  • What about generifying an object? That is binary compatible as long as the type bound is the same as the old type because the type variables are erased to their bounds at compile time. But if the type bound is relaxed, then it’d cause methods to have different signatures and cause a link failure.
  • Making a previously nullable parameter non-nullable is binary compatible and source compatible in Java but not in Kotlin, and not backwards compatible because an API user may have been passing in null. But if you know that this is impossible e.g. because null was previously undefined or crashed in a different way anyway, then it can still be a compatible change in some technical sense.

And so on and so on.

The more I think about this topic the more I want to write a KEEP for using invokedynamic and have a crack at implementing it. The rules are too hard for developers to reliably memorise, even once written down. Most programming doesn’t require you to think about this - only when working with app containers these days (as bumping a dependency in gradle will cause a recompile anyway).

I don’t want to have to write tools to scan JARs to find backwards incompatible changes post-hoc when we could just obliterate the problem entirely with a better compilation strategy.

15

Ok. A second take on the rule: Adding stuff to existing classes generally does not break ABI, changing stuff does.

There are lots of details, that is for sure. Generification is complex topic in particular, because in order to figure out whether it is going to affect ABI you need to really understand how erasure works. Changing nullability you sometimes can get away with, but not if it concerns “primitive” types (which is purely a low-level JVM concept). I wonder if it is even possible to specify all of that in an understandable way.

I would rather approach that from a completely different angle: specify a list of simple changes (aka “adding stuff”) that are guaranteed not to break ABI and leave everything else unspecified, as this kind of specification is going to be tightly-bound to the details of Kotlin-to-JVM byte code translation process anyway.

1 Like