Maintaining binary compatibility is an important aspect of evolving libraries, especially those libraries that other libraries depend on. Why is it important? Because it’s not an uncommon situation when every library brings a different version of the same common dependency to the classpath. This situation is known as [JAR hell] (Java Classloader - Wikipedia).
Contemporary build systems such as Maven or Gradle tend to solve the JAR hell problem by only allowing a single version of the same artifact to get to the classpath. But now you have to decide which version it should be to satisfy all the dependencies. If the library maintains the property of being binary backward compatible, you can safely choose the latest version and have it all sorted out.
Kotlin Standard Library is one of the libraries used ubiquitously as a dependency of Kotlin applications and libraries.
Therefore, we have composed a list of rules describing what can be changed in a library without breaking binary compatibility and a tool binary-compatibility-validator which validates that the library changes adhere to these rules.
One of the changes that is safe from the binary compatibility standpoint is adding a non-abstract method to an existing class. But while it’s safe when a single latest version of the library is kept in the classpath, it can bite hard when there are several versions of the library available. Code that depends on the new version of the library and expects the new method being available can get the class loaded from the old version instead without the method required. A NoSuchMethodError is a common symptom of this situation.
Our original plan was to do our best not to add methods to the existing classes in the Standard Library and then we could live even in the situation of JAR hell. You’d ask then how are we going to add new top-level and extension functions, because they are exactly the new methods in the existing classes? Well, we were going to add these functions to new files so that every new file would be compiled into a new class. That would result in the same experience while consuming those functions from Kotlin, and quite worse experience with Java, as one would have to deal with rather artificial class file names like say Collections12Kt, where ‘12’ stands for Kotlin 1.2 — the version of Kotlin when that file was introduced.
However it appeared that NoSuchMethodError when calling public API is not the worst thing that can happen in JAR hell. When a class can be loaded from an arbitrary artifact version, changes in a private part of API are no longer safe. Imagine we have used new private method in an internal class to implement new public API. Now when the old class version is loaded we’re again missing that method. A more subtle problem can appear when the method is still in place, but its contract is changed.
Therefore in the situation with multiple versions of the library available, all its private API becomes effectively public from the standpoint of binary compatibility, and now we would have to consider not only binary compatibility between the library and other libraries depending on it, but also the compatibility between parts of the library itself.
That’s why we have decided to abandon this plan, and introduce new methods in existing classes without hesitation. If you have any considerations why we shouldn’t do it this way, feel free to share them.