Announcing kotlinx-metadata-jvm library for reading/modifying metadata of Kotlin/JVM class files

Libraries
21

@xdenss Could you please describe process of removing @Metadata annotation with R8?
Just added in gradle.properties entries to support R8 android.enableR8=true.
Also tried with android.enableR8.fullMode=true, but metadata annotation is still visible in decompiled code.

22

kotlinx-metadata-jvm 0.1.0 has been published on Bintray and will be available in Maven Central soon.

The highlight of this release is the new value-based API (KT-26602) which is now the preferred way to work with the metadata instead of visitors. Take a look at an updated ReadMe and/or an example test case to learn more.

For example, here’s an excerpt from that test case that creates class metadata from scratch:

    val klass = KmClass().apply {
        name = "Hello"
        flags = flagsOf(Flag.IS_PUBLIC)
        constructors += KmConstructor(flagsOf(Flag.IS_PUBLIC, Flag.Constructor.IS_PRIMARY)).apply {
            signature = JvmMethodSignature("<init>", "()V")
        }
        functions += KmFunction(flagsOf(Flag.IS_PUBLIC, Flag.Function.IS_DECLARATION), "hello").apply {
            returnType = KmType(flagsOf()).apply {
                classifier = KmClassifier.Class("kotlin/String")
            }
            signature = JvmMethodSignature("hello", "()Ljava/lang/String;")
        }
    }
1 Like
23

@udalov the new API is great, but I was wondering why the approach of all simple classes with mutable variables was taken rather than making them immutable data classes? Taking that same example snippet

val klass = KmClass(
    name = "Hello",
    flags = flagsOf(Flag.IS_PUBLIC),
    constructors = listOf(
        KmConstructor(
            flags = flagsOf(Flag.IS_PUBLIC, Flag.Constructor.IS_PRIMARY),
            signature = JvmMethodSignature("<init>", "()V")
        )
    ),
    functions = listOf(
        KmFunction(
            flags = flagsOf(Flag.IS_PUBLIC, Flag.Function.IS_DECLARATION),
            name = "hello",
            returnType = KmType(
                flags = flagsOf(),
                classifier = KmClassifier.Class("kotlin/String")
            )
            signature = JvmMethodSignature("hello", "()Ljava/lang/String;")
        )
}

Later mutating could be done safely with data class copy functions instead then

24

A separate question - would you be open to a PR with a non-ServiceLoader initialization option? As the library is not stable yet, it’s not safe to ship as a direct dependency yet in its current form. I’ve been testing out shading it here, but have had enough issues with getting service rewriting to work correctly that I’m hoping you’d be open to a non-service solution instead.

25

One more separate question (sorry if this the wrong place, since this isn’t a separate kotlinx repo I can’t file these as separate issues)

Is it possible to distinguish between receiver types and non-receiver types? For example: I don’t currently see a way to differentiate between String.() -> Unit and (String) -> Unit as they’re both just Function1<String, Unit> as far as metadata is concerned

26

Hi! Sorry for a late response.
To reproduce the R8 issue I implemented Kotlin class with reflection.
For example:

This example works fine with Proguard but crashed with R8 with access exception. Without Metadata there is no information about getters for vals.
In R8 sources I found line where metadata is used to collect additional info for desugaring (and maybe obfuscation) and then just remove it. There was following code (at least month ago):
snippet from src/main/java/com/android/tools/r8/naming/ClassNameMinifier.java

1 Like
27

@hzsweers Thanks for the feedback!

I was wondering why the approach of all simple classes with mutable variables was taken rather than making them immutable data classes?

One of the obstacles is the fact that we plan to extend the library to other platforms in the future, thus the API of common classes cannot refer to JVM specifics (such as signature). Currently all JVM extensions are done via Kotlin extension properties and this design seems to scale well in the presence of other platforms.

Another is performance. I agree that data classes seem nicer, but not that much nicer to justify the additional cost of all the allocations caused by mutations, and all the extra equals/hashCode/toString/component methods in the bytecode. We’re still aiming for kotlinx-metadata-jvm to be used in tools such as bytecode obfuscators and optimizers that are supposed to work fast on big codebases.

A separate question - would you be open to a PR with a non-ServiceLoader initialization option?

Yes, although I’m wondering what exactly do you have in mind. ServiceLoader seems to be the preferred way of simple DI (especially in light of the Java module system) and it should be handled properly by the Shadow plugin. In any case, let’s not discuss the details here – if you have an idea, please send a PR and I’ll take a look!

Is it possible to distinguish between receiver types and non-receiver types? For example: I don’t currently see a way to differentiate between String.() -> Unit and (String) -> Unit as they’re both just Function1<String, Unit> as far as metadata is concerned

The way this is handled in the language is by a type annotation ExtensionFunctionType. So the type String.() -> Unit is actually the same as @ExtensionFunctionType (String) -> Unit. You can find type annotations via the KmType.annotations extension property.

28

@udalov Ahhh perfect, didn’t know about the extension properties. Worked like a charm, and I’ve been tinkering with writing an immutable layer over the mutable one for IR use. The mutability/visitor reason makes sense with that context :+1:.

I’ll think on the serviceloader init alternative idea and see what I can do!

29

Is it possible to read file annotations from metadata?

1 Like
30

No, annotations are not stored in the metadata and kotlinx-metadata-jvm doesn’t provide a way to read them. The only workaround is to read them directly by external means, such as from the bytecode, via reflection or annotation processing API. File annotations specifically are generated on the corresponding file facade class, if it exists.

We do plan to provide access to annotations on classes and members (see the comment above and KT-31857), but likely not as a part of the core kotilnx-metadata-jvm library.

1 Like
31

Hello, (sorry if question isn’t directly relevant)

  • I would Like to get KmClass or kClass or just kotlin full name of
    the kclass parameter in my annotation inside my annotation processor
    I read above 2 comments This & This that kotlin.Metadata don’t hold annotations info
    but this can be done by external means So can you please tell me how to achieve
    my goal by external means because I don’t understand.

  • For more clarification take a look at the code below

// Another developer’s code using my library
@MyAnn(kClass = List::class)

// My annotation processor code
val myAnn = element.getAnnotationsByType(MyAnn::class.java).first()

// how to get KmClass or KClass or just String of full kotlin class name
// using myAnn.kClass

// ALSO Note what i currently can do is get it in a TypeMirror instance
// but that has a problem for ex. it shows above example as
// java.util.List instead of kotlin.collections.List

Thanks in Advance.

32

@MohamedAlaaEldin636 In your example, myAnn.kClass should return an instance of KClass, which has a qualified name and lots of other info. So, myAnn.kClass.qualifiedName should return kotlin.collections.List.

If you try this and still have troubles, please create a new topic on this forum instead of this one, since it’s not directly related to kotlinx-metadata-jvm.

1 Like
33

Hey @udalov would it be possible to have a description of what the metadata needs in order to peform reflection? Im writing an obfuscator and its great to have this library, but I think it needs a guide accompanying it. From my brief glances at it, it seems that I should simply change data2 strings to their mapped equivalents. Are there any other fields in the annotation I need to change?

34

@cookiedragon234 With an exception of Kotlin standard library, data1 and data2 are the only properties used meaningfully in kotlin-reflect. However, and maybe I’m not interpreting your question correctly, but it’s not recommended to just change strings in data2 (you wouldn’t need the kotlinx-metadata-jvm library for that anyway). It’s just a string table with all the names used in the metadata, so the same string may refer to multiple names, some of which are going to be obfuscated but some aren’t.

For example, if you have a class A and a function A declared or referenced in the same file, there will be one entry "A" in data2. If the obfuscator then decides to rename the class (but not the function!) to B, it’s incorrect to just change the entry in data2 to "B" since the function would be renamed too as a side effect.

kotlinx-metadata-jvm abstracts that away; you shouldn’t deal with the string table anymore. If you read such class to a KmClass instance, change its name and serialize it back to a .class file, you’ll observe that only the class is renamed. Under the hood, the library will transform the string table so that there’s an entry for the class name "B" and the entry for the unchanged function name "A".

2 Likes
35

Hello,

First of all, congrats on this library. I’m building a code generator tool that plugs into the Java annotation processor API and this library bridges the gap nicely between the two worlds.

Are there plans to continue to support this library in the long run? There hasn’t been a new release for quite some time now. I know that there are plans for a public kotlin compiler plugin API and I’m not sure if this compiler metadata will be deprecated in favour of something new.

36

Hi @pak3nuh,

Yes, kotlinx-metadata-jvm is fully supported and there are no plans to change it. The reason why there are no releases right now has to do with the fact that the library is basically almost feature complete, so we’ve been focusing on releasing Kotlin 1.4 instead. If you have any particular issues in mind, feel free to report them to our YouTrack.

Note that the compiler plugin API effort is not related to kotlinx-metadata-jvm because the latter can be used in contexts where the Kotlin compiler is not being run, i.e. different bytecode processors such as obfuscators, or runtime introspection.

1 Like
37

kotlinx-metadata-jvm 0.2.0 has been released.

The major breaking change in this release is that it no longer supports writing class metadata of versions earlier than 1.4. Reading metadata of old format will continue working fine, but writing it had a complicated issue that could result in compilation errors in dependent code, so it was decided to report an error if metadata version is less than 1.4. You can still use older kotlinx-metadata-jvm versions for writing pre-1.4 class files, however please read the mentioned issue and beware of the fact that older versions also suffered from it.

Other than that, this release finally brings the full support of metadata produced by Kotlin 1.4 by adding a new flag for fun interfaces, and the new metadata for optional annotation classes in multiplatform projects.

Full changelog:

  • KT-41011 Using KotlinClassMetadata.Class.Writer with metadata version < 1.4 will write incorrect version requirement table
    • Breaking change: KotlinClassMetadata.*.Writer.write throws exception on metadataVersion earlier than 1.4.0.
      Note: metadata of version 1.4 is readable by Kotlin compiler/reflection of versions 1.3 and later.
  • Breaking change: KotlinClassMetadata.*.Writer.write no longer accept bytecodeVersion.
  • KT-42429 Wrong interpretation of Flag.Constructor.IS_PRIMARY
    • Breaking change: Flag.Constructor.IS_PRIMARY is deprecated, use Flag.Constructor.IS_SECONDARY instead
  • KT-37421 Add Flag.Class.IS_FUN for functional interfaces
  • Add KmModule.optionalAnnotationClasses for the new scheme of compilation of OptionalExpectation annotations in multiplatform projects (KT-38652)
1 Like
38

kotlinx-metadata-jvm 0.3.0 has been released.

Changelog:

  • Update to Kotlin 1.5 with metadata version 1.5.
    Note: metadata of version 1.5 is readable by Kotlin compiler/reflection of versions 1.4 and later.
  • Breaking change: improve API of annotation arguments.
    KmAnnotationArgument doesn’t have val value: T anymore, it was moved to a subclass named KmAnnotationArgument.LiteralValue<T>.
    The property value is:
    • renamed to annotation in AnnotationValue
    • renamed to elements in ArrayValue
    • removed in favor of enumClassName/enumEntryName in EnumValue
    • removed in favor of className/arrayDimensionCount in KClassValue
    • changed type from signed to unsigned integer types in UByteValue, UShortValue, UIntValue, ULongValue
  • KT-44783 Add Flag.IS_VALUE for value classes
    • Breaking change: Flag.IS_INLINE is deprecated, use Flag.IS_VALUE instead
  • Breaking change: deprecate KotlinClassHeader.bytecodeVersion and KotlinClassHeader’s constructor that takes a bytecode version array.
    Related to 'KT-41758`.
  • KT-45594 KClass annotation argument containing array of classes is not read/written correctly
  • KT-45635 Add underlying property name & type for inline classes
    • See KmClass.inlineClassUnderlyingPropertyName/inlineClassUnderlyingType.
1 Like
39

@udalov

I wasn’t sure which way I should post my question, so excuse me here.

Can I assume that the type property of the KmValueParameter retrieved from the KmFunction will not be null?
I have tried with 0.3.0 and the type property seems to return PrimitiveArray (e.g. IntArray) or Array.

I would appreciate it if you could tell me if there is a pattern where this property is null.

p.s.

I find this library very interesting.
I am trying to see if I can apply this library to various projects.
Thank you.

40

@wrongwrong Thanks! Your observation is correct, KmValueParameter.type should never be null. I’ve reported an issue at https://youtrack.jetbrains.com/issue/KT-48965.

1 Like