Can't add annotation by compiler plugin

I’m trying to create kotlin compiler plugin that will parse all the doc comments and generate @Schema annotation for them. So what I’m doing now:

@Suppress("UNCHECKED_CAST")
class OpenApiCommentsGenerationTransformer(
    private val pluginContext: IrPluginContext,
    private val msgs: MessageCollector,
) : IrElementTransformerVoidWithContext() {
    @ObsoleteDescriptorBasedAPI
    override fun visitClassNew(declaration: IrClass): IrStatement {
        val schema = IrSingleStatementBuilder(pluginContext, Scope(declaration.symbol), -1, -1).build {
            // schema generated here
        }

        declaration.annotations += schema
        return super.visitClassNew(declaration)
    }
}

But this works only when class already has some annotation. E.g. with this class

@MyAnn
class TestObj

I get next bytecode generated:

@comments.to.openapi.MyAnn @io.swagger.v3.oas.annotations.media.Schema public final class TestObj

So both annotations are present. But if I have next declaration

class TestObj

I’m getting:

public final class TestObj

declaration.dump shows that Schema annotation is generated in both cases. But for some reason it’s not written to the result .class file. The same story for properties.