KMP iOS UIImage Memory Leak

I’ve been working on an application which uses Kotlin Multiplatform as the shared business logic which handles image processing. for iOS, I send in multiple UIImage’s to a kotlin defined function, and then return the image once the acceptance criteria has been met. But right now when I send the UIImage’s into the Kotlin defined function, the memory on iOS will rapidly climb until the app crashes. Even when I remove all processing and just pass the UIImage into the kotlin function, there are identifiable memory leaks and the app will crash eventually due to memory overload.

In swift, I have a camera view class that creates a videostream for the user. I then extract each frame and convert it to a UIImage:

// CameraView.swift
// capture delegate from video output stream
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    DispatchQueue.main.async { [weak self] in
        guard let uiImage = self?.imageFromSampleBuffer(sampleBuffer: sampleBuffer) else { return }
        connection.videoOrientation = .portrait
        // delegate to owning viewModel
        self?.delegate.onFrameCaptured(image: uiImage)
    }
}
// conversion function to UIImage
func imageFromSampleBuffer(sampleBuffer : CMSampleBuffer) -> UIImage? {
        guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return nil }
        let ciImage = CIImage(cvPixelBuffer: imageBuffer)
        guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else { return nil }
        return UIImage(cgImage: cgImage, scale: 1.0, orientation: .up)
}
    

Inside the owning viewModel, I simply pass this UIImage from the delegate responder function down to the Kotlin defined function.

// ImageCaptureViewModel.swift
lazy var documentDelegate = DocumentFrontDelegateImpl(identify:self)
func onFrameCaptured(image: UIImage) {
        if (captureState != .autoFailed) {
            // Kotlin-defined function
            documentDelegate.process(image: image)
        }
    }

Inside my kotlin defined function, I have a print debug statement showing the correct function is being called.

// DocumentFrontDelegateIml.kt
open class DocumentFrontDelegateImpl(private val identify: IDocumentDelegateImpl) {

    fun process(image: UIImage) {
        println("image received in KMP")
    }
}

I am not doing any processing with the images, but still the application crashes within 15 seconds. I am sending ~30 frames per second into this kotlin function, and the memory usage climbs rapidly before crashing the app. I’ve also tested it out with only sending 1 image down to Kotlin, and that image stays in memory forever.

Using instruments I was able to see the memory leak, and notice that Core Graphics has 2 retain counts for each image, but only 1 release. I.E. each image stays in memory for the duration of the application.

Has anyone experienced anything similar to this? I found that a CGImage will have the same problem with memory. The references are never fully garbage collected and stay in memory forever.

Here is build.gradle.kt file for how I configure the iOS framework that gets built

kotlin {
  val framework = XCFramework()
      ios {}
      iosSimulatorArm64 {}
}

I am currently using the experimental memory model.

# Multiplatform
kotlin.native.binary.memoryModel=experimental

Any insight would be appreciated!! This memory leak is a huge blocker for me right now and I can’t find any way around it.

I don’t immediately have a solution, so I am just guessing a bit:
Looks like the one release operation is triggered by Kotlin. Now the question is: There are two retain request, but does Kotlin get both references/ is responsible for both? Shouldn’t one of them be Swift-only, basically a local variable where the image is stored before passing to Kotlin? But if Kotlin is only responsible for one reference and releases one reference, is the memory leak probably in the Swift part?

Can you please post the memLeakProfile if you comment out the call to Kotlin (without changing anything else)?

    // documentDelegate.process(image: image)

Sure thing. here is the memleakProfile when I comment out that call to the kotlin defined function. The memory leaks aren’t present anymore and the application will run normally.

I think the swift retain count is getting properly released, but the reference created when passing to the kotlin function is not.

Very well. Can you please check what happens when you remove the attribute/property from the Kotlin class:

open class DocumentFrontDelegateImpl {

    fun process(image: UIImage) {
        println("image received in KMP")
    }
}

and at call site:

lazy var documentDelegate = DocumentFrontDelegateImpl()

Out of curiosity, I would also like to take a look into the call stack of the retain request.

No luck when removing the attribute in the Kotlin class. Memory still overloads and crashes. Here is the stack trace for one of the leaked images:

Malloc:

CFRetain:

CFRelease:

Some observations; maybe one of them leads to something:

  • malloc and retain are requested from DocumentCaptureFrontViewModel.swift, but your shown source code is from ImageCaptureViewModel.swift.

  • In both requests, it seems that process calls something on the companion object of the class ImageParcel, but in your source code process does nothing (except printing a static string).
    → Is ImageParcel one of your local classes or is this automatically created by the Kotlin interoperability?
    → Just in case it is your local class: How/Why is it called here?

  • The malloc request calls CGImageCreateWithImageInRect which says in its documentation (Apple Developer Documentation)

The resulting image retains a reference to the original image, which means you may release the original image after calling this function. In Swift, you do not need to release the original image reference explicitly.

If the ImageParcel and cropping calls are part of the automatic interoperability (conversion?), then it surely looks like two references are created. Since Kotlin has a full-automatic garbage collector and Swift does not (only ARC), it is pretty much imaginable that the developer of the interoperability forgot about releasing the reference - being accustomed to the simple comfort of Kotlin instead of the more tedious half-manual memory management of Swift. At least, I think it’s possible. Still, I hope the points above maybe lead to something.

Thanks for the feedback @tlin47 !!

addressing your points:

  • ImageCaptureViewModel.swift was a generic name I put since we have 3 separate viewModels responsible for calling the same kotlin function. In this test case, I was using DocumentCaptureFrontViewModel.swift → so this is correct
  • ImageParcel is our data structure class for sending the images for processing. In the Instruments profiling pictures I sent, I had hooked up the processing again to debug further (sorry for the confusion). So these function calls were also expected, not part of the automatic interoperability.
  • ImageParcel and CropToSubject are called within the kotlin defined process function:
fun process(image: UIImage) {
        delegate.sendImage(IImageParcel.of(image).cropToSubject())
    }

The processing of the images [sendImage()] is written in platform generic code. Is there a way we can manually call to release the references of the images passed in once finished??

Side note**
While I was debugging yesterday I found something interesting:

  • when removing the sendImage() function in Kotlin, the memory usage is still very high and will eventually lead to a crash. But when I limit the frames sent into kotlin (down to 10 per second), the memory seems to be properly garbage collected at certain intervals.
  • And when I add that sendImage() function back into kotlin, with the 10 fps limiter, the app will overload the memory and crash. So I think you’re right with the refs being created in kotlin and then not properly released.

is there a way to modify how frequent the garbage collector runs ??

That is a good question. I think it would be interesting to know how the function looks like that contains ImageCropping, line 42.

I haven’t done this manually. I know that in Java, one can use System.gc() to request the garbage collector to do his work now (although it is no guarantee that it will actually free up everything immediately). I am not sure how the Kotlin way is for it.

I found a leak!
it was in my imageCropping function:

return UIImage(CGImageCreateWithImageInRect(source.CGImage, cropRectangle))

this function creates a CGImageRef that, when unhandled, will sit in memory and creates leaks.

Solution:

val imageRef: CGImageRef? = CGImageCreateWithImageInRect(source.CGImage, cropRectangle)
val uiImage = UIImage(imageRef)
CGImageRelease(imageRef)
return uiImage

I needed to manually release the CGImageRef before returning the UIImage. After profiling, all leaks are gone and the application runs normally :slight_smile:

@tlin47 I sill have an issue with the frames I’m sending into the kotlin defined function (sending 30 frames a second will overload the memory and crash the app). I will create a new topic about this issue as this one has been resolved.

1 Like