Proposal: Kotlin multi-release jar support

I propose some minor changes to Kotlin language to support multi-release jars. The proposal is based on annotations, and it will require some compiler changes and syntax improvements, along with tooling changes (and things could be a bit hard here, since some tool design assumptions could be broken). This is currently a very rough idea, but I think that it is a good enough for releasing now. I think it could be very useful for library developers for JVM platform. The proposal could be reformatted as KEEP by anyone at any time if needed, I’ve posted it here to start an initial discussion of the general idea.

The proposal introduces a new annotation MultiRelease on classes (and top-level objects), type members, and imports. The name of annotation is under consideration. It could be JvmMultiRelease, for example.

The class annotation (or object):

@MultiRelease(since = “8”, untilExclusive = “11”)
class MyClass {
    …
}

This annotation indicates that class supports multi-release compilation. The property “since” defaults to the empty string, that means module java version. The property “untilExclusive” defaults to the empty string as well, and it mean “for all future versions”.

For multi-release compilation, if base Java version of the module is lower than the specified in annotation, the file is compiled to the corresponding multi-release root in jar file.

Multiple definitions of the same class with different specified versions might be present the source tree. By convention, if different files are used, the files with version higher than the base module version will have “_” suffix in the file name, like “MyClass_11.kt”.

If the until is version is not blank, the version is supposed to be used until the corresponding version.

If the class is annotated as multi-release, it could have members annotated with multi-release annotation. For each unique version, the class is treated as it would have been a separate multi-release file with the specified members for each unique version mentioned. The class is compiled as several passes for each unique multi-release annotations. There could be different number of members with different names for each multi-release version.

For example:

@MultiRelease
class FieldProcessor {
      @MultiRelease(since = “8”, untilExclusive = “11”)
      var field : Field;
      @MultiRelease(since = “11”)
      var varHandle: VarHandle;
      …
      fun processValue(object:Object) {
            val fieldValue = internalGetValue(object)
            doSomething(fieldValue)
      }
      @MultiRelease(since = “8”, untilExclusive = “11”)
      private inline fun internalGetValue(object: Object) : Object {
                // use field and reflection
      }
      @MultiRelease(since = “11”)
      private inline fun internalGetValue(object: Object) : Object {
            // use varHandle
      }
      …
}

Annotations are supported only on the member level. If some small expression needs to be multi-release, it could be abstracted as inline method.

When comparing versions, the class multi-release annotation wins, so if we have version 11 class with member of version 17 and class for version 16. The class with the version 16 wins. However, during compilation a warning (or maybe even error) is reported. If a module has version 11, and there is multi-release annotation with version lower than 11 (in “since” or “untilExclusive”), the compiler should produce warning as well.

The annotation could be also supported for the package, it will mark all classes in this package and sub-packages as multi-release. The annotation on the class within package, will override package annotation values.

The annotation could be also attached to import statement. This might be extension to grammar. In that case import will behave according to the same rules of version availability. Affected multi-release imports will be available only from multi-release code. Normal code will not be affected. Annotations on imports could be useful for other purposes in the future, so it might be not bad feature overall. There might be special Kotlin annotation (for example, @JvmImportAnnotation) in parallel to @Target that would specify that this source annotation is attachable to imports.

The annotation has source retention, as it only affects compilation, and after that the result is a standard multi-release jar with sources.

Non-multi-release code will see only the code the base module version. While multi-release classes could have inline functions internally, the other classes (whether multi-release or not) could only refer to non-inline methods. This restriction could be lifted in future, if there will be a sane implementation strategy for this. With mixed compilation, Java code will use only base version of Kotlin sources, unless Java will start to support multi-release too in some form.

The compilation process will be like the following:

  • Compile the base version and put classes to the root classes directory.
  • Compile the next multi-release versions in sequence putting files to release-specific directory within classes directory.

Options:

  • Release numbers could be supplied as integers or even enum literals. Strings are just more future-proof. Using some constants would not hurt in any case.
  • The additional property “untilInclusive” might be specified for specifying last good version for code instead of the first bad (but this is possibly overkill).
  • Use comments instead of annotations for multi-release imports (I do not like this idea).
  • Compiler could generate errors, if there is any intersection of intervals.
  • Non-top-level objects might support this annotation too for members. In that case the container class or object should be annotated as multi-release.
  • Top-level functions could be supported as well. This might have more tricky implications for performance of IDE and Compilers due to their number. I think that this option should be avoided for the initial version, and to be evaluated later. Nothing forbids such global function to refer multi-release objects.

JDK Versions Considerations

When compiling, the compiler will require all JDK versions mentioned in annotations configured.

If version is not available, it might sense use pre-build index, that could be downloaded as separate optional dependency for compiler. The prebuilt index is used only for past versions relative to supplied JDK version. So, version 11 index is available with JDK 16, but the version 16 index is not available for JDK version 11. This is an optional feature that might be implemented much later. The first versions of the feature could just proceed with requiring actual JDKs to be supplied.

The index will contain only minimal availability information, and it could be avoided if the JDKs are explicitly supplied.

It is expected, that multi-release versions in the code will follow the current LTS pattern: [8-]11(-17)[-possiblyWhateverPreviousJustInCase]-whateverCurrent. So, the number of versions to be considered usually will be small.

IDE Considerations

In IDE for multi-release modules, a number of tags will appear above source, for “all versions” and for each unique multi-release version in the file. By default, “all versions” is selected and the background is white (or dark), if tag is selected, the non-active pieces of the code are marked as grey (or whatever is background for the dark scheme). This will allow to focus on the specific version when needed.

For files with multi-release code, it is also possible to display version tags attached to files on the source tree in project view.

The member resolution is done for each version at the same time. Errors are highlighted with tags of versions where they present.

The module java version and supplied JDK form version range supported for the module. IDE might need to be modified to support additional “informational” JDKs for module, for intermediate versions.

If multiple JDKs are configured (8, 11, 16 etc.), it would be useful to allow running the tests on multiple configured JDKs by default, as test also could be multi-release.

The debugger might need to be aware of multi-release nature of the module for expression evaluation. However, the debugger needs to be aware only about actually running version.

Performance Considerations

The IDE and compiler cost will be higher only for the specific multi-release files. Other files are not affected, as they refer to base version of class. But if these files were written using multi-module projects and some jar post-processing magic, the cost would have been higher. Cost of annotated files will increase with each unique version specified, but that cost would have been paid anyway in case of multiple files.

The runtime performance is not affected, because if one needs multi-release file, one has to create anyway.

Cognitive Load Considerations

The proposal has lower cognitive load on the human than the current multi-release trickery. The code is concentrated in a single place, rather than spread over all source tree. Only changed aspects are specified and it is easier to understand. There is no need to compare files from different directories to understand a difference. Also, it is less likely to miss copying a piece of code to the copy related to other release. A completely new class version is needed only when it is a complete rewrite of the class.

Expected Effect on Kotlin

The multi-release file support could make Kotlin attractive for writing libraries or products that really need multi-release. The start could be just a mixed compilation with transformation of multi-release parts to Kotlin, while keeping rest in Java.

This will also speed up Java version adoption in the libraries, because it will be much simpler to have an optional support for new java feature without introducing hard to manage build magic.

Possible Future Development

This proposal is focused only on multi-release support for JVM. A similar approach could be used for multi-platform source trees. It will result in different classes compiled for different platforms. Within platform, it is possible to annotate with platform version as well (for example, android releases), it will result in platform-dependent behavior (for example, multiple output files, or even some browser detection code that will serve a correct version for a browser).

It is also possible to use this feature when implemented for release-dependent compilation. For example, compiling data classes in the way compatible to Java records for corresponding JVM release version. It would be some kind of implicit multi-release jars.

The multiple platforms and platform releases are fact of life, and I think that we should acknowledge this in the language, rather than ignore. This issue is very important for reusable code libraries and components. So, this proposal just a further application of DRY principle for Kotlin.

I suspect that dumping all code in the same source file can produce larger classes harder to understand.

Having a better support for multi-release JAR file would be nice, but I think that this isn’t the way to go.

1 Like

If the file will be too larger with too much differences, there is option to have separate files with classes with the same name, but different MultiRelease annotations. Like

MyClass.kt
@MultiRelease(untilExclusive = 11)
class MyClass {

}
MyClass_11.kt
@MultiRelease(since = 11)
class MyClass {

}

The first class will compile to root of the jar, other class will compiler for release-specific directory in jar. And this option is mentioned in the proposal.

Member annotation approach is for cases with few isolated differences inside one class, or for adding members, which signature depends on new type (for example, VarHandle specific method is added to class that have mostly Java8 type members).

Yes,
I understood that, but it does not affect my previous consideration.

The support for JEP 238 should be enough.

I do not see how it is larger. If we split source with 10 method, where only one method is different. We could have one class with 11 methods, or two classes with 20 methods total. And because there is duplication of the code in 9 methods, we need to maitain a copy, so changes done in one source have to be transferred to other source. Refactoring also will not notice a copy, because relationship is not obvious.

If we have a seaparate copy in the additional module, we will have an additional code size due to direcories, build files, and depenendcies. This is also infrastructure code that has to be maitained.

In some case it is possible to refactor this single method to separate adapter class, but this will increase total complexity of the application as well.

The annotation-based approach here looks a reasonable minium of efforts on the part of the library developer that I see. And if we have a single jar file output for the module, it is reasonable to produce it from a single source tree in the scope of single module, without jar post processing magic.

You wrote about the current limitations of JEP 238 and I agree with you that the overall developers’ experience can be improved.
On another side, these current limitations force developers to reorganize the code to contain the differences.

I worry that this kind of solution, easy to implement, does not scale well on the larger codebase, especially if multiple implementations live together in the same file.

It is not that we know exactly until we try. I think more feedback from library developers is needed here.

The current way does not scale well too, as changes between version are not highlighted clearly and are implicit. The diff is needed to discover changes and to interpret them. This approach make release-releated changes explicit, so one does not have to infer or guess them. So from what I see, the cognitive load is strictly less than for multi-module magic apporach. And in the current approach, one even do not know by looking at a class whether the class is multi-release or not, so changes to parallel versions could be just forgotten.

Also changes like Field to VarHandle (in example) will be likely rare. Most of the changes will be likely addition of new methods to support new data types, or implementation changes that switch to API with better performance.

It is already possible when using Gradle as build tool and there is no need to modify how .jar works.

In this article it is explained how Gradle can handle variant aware publications showing how it would be useful to address the many versions of Guava (for JRE 8, 11, Android, etc…).

In the article it is shown how to create a plugin can effectively address which version of Guava you should import based on you build configuration. Check it out!

Also keep in mind that the Gradle approach to build variants is not tight to any specific language, it could be used wherever!

Multi-release jar is a bit different beast than multiple releases of jar file. The support for different versions is compiled into a single jar. It is supported by gradle in some way. The problem is that it is a way too implicit, and leads to code duplication, unclear dependencies, and other problems interesting problems.