Contributing a PR to implement better ABI stability for default parameters

1

Hello JetBrains,

My company maintains a large API and ABI stable platform written entirely in Kotlin. One feature causing us pain at the moment is that adding new default parameters to a method, even when annotated with @JvmOverloads, is ABI stable only for Java clients - not Kotlin clients. The latter will break because the method call is emitted as a call to the version of the method that takes all parameters with the bitmask, and the overloads that are generated don’t provide that bitmask.

We’ve been able to work around it for now with a variety of increasingly horrible hacks, but this will pollute our codebase over time.

This is probably not too hard to fix in the compiler directly, unless there are subtleties we’ve missed. If we contribute a PR to make the compiler emit additional synthetic methods to class files to keep old binaries working in the presence of new params, will that be accepted? Is there anything we should watch out for - e.g. are you planning a rewrite of relevant areas?

3 Likes
2

So if I get it correct, you want to keep binary compatibility with the old code that called the method without that new parameter. In that case you can keep the old method and introduce new method with an additional optional parameter besides. The old method should be annotated with Deprecated(level = HIDDEN) in order to hide it from Kotlin clients.

3 Likes
3

There are such hacks available yes but it’s easy to forget. We rely on a custom built ABI checking tool today to find such problems.

It does not seem like a big leap to say that if something is tagged with @JvmOverloads, we could generate overloads for the fewer parameter versions that support old clients too. The bytecode cost is trivial and will not incur issues if they’re never executed. Otherwise we have to remember to always introduce such things, and they clutter the source code.

More generally I feel like source/binary compatibility is one of those low level issues managed languages should really abstract us from. It’s very tricky, the rules are subtle and complex and many developers aren’t used to them. A bit like manual memory management, it’d be nice to just hide it.

2 Likes
4

Hi @mikehearn ,

From my perspective your issue is 100% legitimate. But @JvmOverloads does not solve the problem for all cases. Kotlin calls the other one because of named parameters. What is needed is to generate (as @ilya.gorbunov suggested) another version that can handle the named parameters. Probably the easiest approach to solve this is to annotate each (new) parameter, but it is somewhat limited in the case that you have more than 2 versions (old and new). Perhaps the annotation could optionally have an int parameter to make it work. Overall this would look like:

fun updatedVersion(
    param1: Int,
    param2: Int,
    @NewApi(1) param3: Int,
    @NewApi(2) param4: Int
)

This would generate 3 versions of the Kotlin ABI compatible implementation. If however you maintain parameter order it would be technically feasible to just generate all possible overloads as you suggest and leave reordering to the non-automatic case.

Btw. this feels like the ideal use case for a compiler plugin (at least in testing). Of course that is not there yet.

1 Like
5

Let’s split this into two issues:

  1. The immediate tactical issue that we need a fix for. We are polluting our codebase with lots of methods that exist for no other reason that to keep the JVM linker happy, this is the sort of thing in a compiler’s domain. We can tolerate not being able to re-order named parameters for now, we have an ABI checker tool that will catch it if someone tries. So just generating more synthetic methods will help. This is the proposed PR.
  2. The longer term issue. I’d rather not solve this by introducing new annotations or language features … the goal should be to eliminate complexity and obscure requirements on developers, not just make them more visible. It feels to me like the “right” fix for this is to experiment with lifting Kotlin method linkage out of the JVM and into the Kotlin stdlib using invokedynamic, the linker code in the stdlib can then handle all the cases where binary and source linkage don’t match. For instance it can detect that a method would fail to link because of named parameter ordering changes, and then use reflection data to patch things up so the program can proceed. It can handle the case where a parameter type becomes a supertype, thus breaking binary linkage but not source linkage, and patch that up too. And so on and so forth. This is a more sophisticated and invasive change, but it doesn’t have to be too invasive, and making it much easier to build backwards compatible software can only have a positive effect on the whole Kotlin/JVM ecosystem.

But for now I’d be happy with a fix for (1). I’m still waiting for feedback on the tactical compiler-level fix from JetBrains. If we get the go-ahead I’ll schedule it into my team’s backlog to tackle.

2 Likes