Announcing kotlinx-metadata-jvm library for reading/modifying metadata of Kotlin/JVM class files


#1

Please welcome a new library kotlinx-metadata-jvm designed for tools that need to read or modify the Kotlin metadata, such as bytecode obfuscators and annotation processors. With kotlinx-metadata-jvm, it’s possible to make sense of, and modify the data written by the Kotlin compiler in the @kotlin.Metadata annotation on the .class file, and in the .kotlin_module files.

Refer to the ReadMe for a quick introduction:
https://github.com/JetBrains/kotlin/blob/master/libraries/kotlinx-metadata/jvm/ReadMe.md

The library is published under the kotlinx project and the version number is 0.* because we’re not yet sure that the API is optimal and won’t need major refinements. Until the library is released as stable, we reserve the right to perform both source- and binary-breaking changes; however of course, we’ll do our best to minimize migration issues when these occur.

Example usage in a Gradle project:

repositories {
    mavenCentral()
    maven { url "https://kotlin.bintray.com/kotlinx/" }
}

dependencies {
    compile "org.jetbrains.kotlinx:kotlinx-metadata-jvm:0.0.2"
}

I’ll use this forum thread to notify about new versions of kotlinx-metadata-jvm and will be happy to answer any questions about the API or future goals of the library here.


#2

Hey @udalov! I’ve been trying this out in Moshi as well as a prototype for something, but in both cases I’m caught by a strange serviceloader issue where it can’t find any MetadataExtensions. They are visible in the Moshi kotlin/codegen compiler tests, but not when used in another artifact as a kapt dependency. I don’t see anything overtly wrong so not sure if I should file a bug. The crux of the issue is that it seems the only classpath available during kapt processing is the embedded compiler classpath, and I don’t see a way to manually add (or we we should need to) add any custom classpath arguments. Any thoughts?

Moshi work switching to kotlinx-metadata (maven): https://github.com/square/moshi/pull/570
CopyDynamic (gradle): https://github.com/hzsweers/copydynamic


#3

Hi @hzsweers, this looks like an artifact of using maven-shade-plugin; I’ve commented in the moshi issue. If it doesn’t work out anyway, please try disabling the relocation of kotlinx-metadata altogether and see if it works. Otherwise, please report it to our issue tracker and I’ll have a closer look. Thanks!


#4

Yep it still happens even with shading disabled (the copydynamic project linked also doesn’t use shading). Filed here: https://youtrack.jetbrains.net/issue/KT-24881


#5

kotlinx-metadata-jvm 0.0.3 has been published.

What’s new:

  • Support metadata of local delegated properties (see JvmDeclarationContainerExtensionVisitor.visitLocalDelegatedProperty)
  • KT-24881 Use correct class loader in kotlinx-metadata to load MetadataExtensions implementations
  • KT-24945 Relocate package org.jetbrains.kotlin to fix IllegalAccessError in annotation processing

#6

kotlinx-metadata-jvm 0.0.4 has been published.

What’s new:

  • KT-25920 Compile kotlinx-metadata-jvm with JVM target bytecode version 1.6 instead of 1.8
  • KT-25223 Add JvmFunctionExtensionVisitor.visitEnd
  • KT-26188 Do not pass field signature for accessor-only properties

#7

Thanks for all this great work!
I was wondering why is this not published in mavenCentral as I’m using this in a kapt processor and each client that use the processor it force those to add them the maven repository:
maven { url “https://kotlin.bintray.com/kotlinx/” }

This is really annoying as I have to leave as requirement for all libs that use this processor to add this other repository. Any idea how to overtake this?


#8

@juanucc Thanks for the feedback. There’s no real reason why we aren’t publishing to Maven Central. I’ve created an issue: https://youtrack.jetbrains.com/issue/KT-27991


#9

kotlinx-metadata-jvm 0.0.5 has been published.

What’s new:

  • KT-25371 Support unsigned integers in kotlinx-metadata-jvm
  • KT-28682 Wrong character replacement in ClassName.jvmInternalName of kotlinx-metadata-jvm

#10

Hi,

I’m trying to use this kotlinx-metadata-jvm library to edit @kotlin.Metadata annotations after a library has had its bytecode modified using ProGuard. My plan was to read the metadata in, filter out the unwanted elements using KmXxxVisitor objects and then write my updated metadata out via the ASM library. However, these KmXxxVisitor objects don’t appear well-suited to such filtering.

For example, consider KmConstructorVisitor: I need accept() to invoke the visitExtensions() method in order to obtain the constructor’s JvmMethodSignature object, and hence learn which constructor’s metadata I am actually dealing with. However, the visitValueParameters() method has already been invoked by this time, which means that I am no longer in a position to affect any value parameters’ metadata.

Similarly with KmFunctionVisitor, because KmClassVisitor.visitFunction() will only tell me the function’s name and I can only learn the function’s descriptor once KmFunctionVisitor.visitExtensions() has provided me with the JvmMethodSignature. (Function names obviously not being unique due to overloading.)

I could still make this filtering work, provided all visitExtensions() methods were guaranteed to be visited first! However, this clearly isn’t the case at the moment.

Have I missed something fundamental about this kotlinx-metadata-jvm library please? I have been proceeding mainly by reading the KDocs and by trial-and-error, so this is entirely possible.

Thanks for any help here,
Cheers,
Chris Rankin


#11

Hi @chrisr3,

You’re right, the existing visitors are a bit inconvenient for that purpose. We’re planning to provide a more convenient value-based API where you’ll be able to simply obtain all necessary information as a mutable object model (KT-26602).

Could you please clarify though, what exactly do you want to do with the metadata? There could be a way to make this work, for example by splitting the visitor into two visitors, one that does filtering and another that does transformation. The key observation is that all writer visitors in kotlinx-metadata-jvm only write everything they’re supposed to write in visitEnd methods. Therefore, if you collect all the necessary info before visitEnd is called, you’ll be able to use that in visitEnd to prevent the writer’s visitEnd from being called if needed.

In regards to making visitExtensions being called first – this could help in your case, but I’m not sure if it’s the correct solution in general. I’ve described it in more detail in the issue you’ve reported (KT-29140).


#12

What I’m trying to do is “fix up” a Kotlin library that has been shrunk using ProGuard (and other tools) so that it’s safe to compile against. The Kotlin compiler seems to use the @kotlin.Metadata annotations as the Source of Truth for which APIs exist in the bytecode, and so it will happily create references to methods and fields that ProGuard has discarded, which then generate NoSuchMethod and NoSuchField errors at runtime.

I currently have a working solution that involves using the metadata’s ProtoBuf classes directly, but would prefer to “future proof” my code by using a public API rather than an internal one.


#13

I have created a YouTrack tag for kotlinx-metadata-jvm issues for easy reference.