TextToSpeech and coroutines

Android
1

Hello, I am new with Kotlin.
I have a published application and try to re-write some code from Java to Kotlin.
I used my implementation of Promise and Deferred objects (my Deferred allows to resolve inner promise from outside) for tts initialization and tts speaking. My purpose is controlling when initialization/speaking started and ended. I hoped that my code in Kotlin will be simpler.

There is working implementations:

class TextToSpeechManager(private val context: Context, private val locale: Locale) {
    private val tag = "TTS"

    private val utteranceManager = UtteranceManager()
    private var ttsDeferred: Deferred<TextToSpeech> = init()

    private suspend fun get(): TextToSpeech {
        Log.d(tag,"get")
        if (ttsDeferred.isCompletedExceptionally) {
            ttsDeferred = init()
        }
        return ttsDeferred.await()
    }

    private fun init(): Deferred<TextToSpeech> {
        Log.d(tag,"init")
        return async {
            suspendCoroutine<TextToSpeech> { continuation ->
                selfReference<TextToSpeech> {
                    TextToSpeech(this@TextToSpeechManager.context) { status ->
                        Log.d(tag,"init " + status)
                        if (status == TextToSpeech.SUCCESS) {
                            val languageStatus = self.setLanguage(locale)
                            Log.d(tag,"init " + languageStatus)
                            when (languageStatus) {
                                TextToSpeech.LANG_AVAILABLE,
                                TextToSpeech.LANG_COUNTRY_AVAILABLE,
                                TextToSpeech.LANG_COUNTRY_VAR_AVAILABLE -> {
                                    self.setOnUtteranceProgressListener(utteranceManager)
                                    continuation.resume(self)
                                }
                                else -> continuation.resumeWithException(Exception(languageStatus.toString()))
                            }
                        } else {
                            continuation.resumeWithException(Exception(status.toString()))
                        }
                    }
                }
            }
        }
    }

    suspend fun speak(value: String, queueMode: Int) {
        Log.d(tag,"speak " + value)
        val tts = get()
        suspendCoroutine<Unit>{ continuation ->
            val utterance = utteranceManager.create(value, queueMode, continuation)
            speak(tts, utterance)
        }
    }

    private fun speak(tts: TextToSpeech, utterance: Utterance) {
        Log.d(tag,"speak " + utterance.id)
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
            tts.speak(utterance.text, utterance.queueMode, null, utterance.id)
        } else {
            val params = hashMapOf(TextToSpeech.Engine.KEY_PARAM_UTTERANCE_ID to utterance.id)
            @Suppress("DEPRECATION")
            tts.speak(utterance.text, utterance.queueMode, params)
        }
    }
}

class UtteranceManager : UtteranceProgressListener() {
    private val utterances = HashMap<String, Utterance>()
    private var nextUtteranceId = 0L

    private val tag = "UtteranceManager"

    @Synchronized fun create(value: String, queueMode: Int, continuation: Continuation<Unit>): Utterance {
        Log.d(tag,"create " + value)
        val utteranceId = generateUtteranceId()
        val utterance = Utterance(utteranceId, value, queueMode, continuation)
        utterances.put(utteranceId, utterance)
        return utterance
    }

    @Synchronized override fun onStart(utteranceId: String) {
        Log.d(tag,"onStart " + utteranceId)
        val started = utterances.remove(utteranceId) as Utterance
        if (started.queueMode == TextToSpeech.QUEUE_FLUSH) {
            for (flushed in utterances.values) {
                flushed.continuation.resumeWithException(Exception("UTTERANCE_FLUSH"))
            }
            utterances.clear()
        }
        utterances.put(utteranceId, started)
    }

    @Synchronized override fun onDone(utteranceId: String) {
        Log.d(tag,"onDone " + utteranceId)
        val utterance = utterances.remove(utteranceId) as Utterance
        utterance.continuation.resume(Unit)
    }

    @Synchronized override fun onError(utteranceId: String) {
        Log.d(tag,"onError " + utteranceId)
        val utterance = utterances.remove(utteranceId) as Utterance
        utterance.continuation.resumeWithException(Exception("UTTERANCE_ERROR"))
    }

    private fun generateUtteranceId(): String {
        val utteranceId = nextUtteranceId++
        return utteranceId.toString()
    }
}

class Utterance(val id: String, val text: String, val queueMode: Int, val continuation: Continuation<Unit>)

// third-party solution
class SelfReference<T> internal constructor(initializer: SelfReference<T>.() -> T) {
    val self: T by lazy {
        inner ?: throw IllegalStateException("Do not use `self` until `initializer` finishes.")
    }

    private val inner = initializer()
}

fun <T> selfReference(initializer: SelfReference<T>.() -> T): T {
    return SelfReference(initializer).self
}

usage:

val ttsMan = TextToSpeechManager(this, Locale.getDefault())
button.setOnClickListener({
    launch {
        Log.d("APP", "before speak")
        ttsMan.speak(text.text.toString(), TextToSpeech.QUEUE_FLUSH)
        Log.d("APP", "after speak")
    }
})

This code is working as expected. But I don’t like nesting async → suspendCoroutine, selfReference hack.
Is there more elegant solution with Kotlin and coroutines?

2

Futures are considered harmful to good code and Deferred is no exception here. You should simply declare your init function as suspend fun:

private suspend fun init(): TextToSpeech = suspendCoroutine { continuation ->
  ... // all the code you currently have inside suspendCoroutine
}

then you can simply define your tts instance as nullable (instead of ttsDeferred):

private var tts: TextToSpeech? = null

and try init it in your get function:

private suspend fun get(): TextToSpeech {
    Log.d(tag,"get")
    return tts ?: init().also { tts = it } // if init crashes, then tts is left null
}
3

Thank you for response.
Your code is lazier. But the typical problem is very long initialization of TextToSpeech on some Android devices (up to 30 seconds!). In my project I start TTS (create TextToSpeechManager) at Application.onCreate and hold deferred to prevent creations many TextToSpeech objects (it’s like “deferred singleton”).

Your variant doesn’t prevent creations. During long initialization tts-field will be null. If I click on button several times - it starts a lot of initializations. Just place “delay” in init fun.

The second problem. If I wait when initialization finishes, I click the button several times and after that I get exception:
Activity ***.MainActivity has leaked ServiceConnection android.speech.tts.TextToSpeech$Connection@7dc4316 that was originally bound here

4

First of all, to make sure your don’t have any leaks to your activity, you should apply an appropriate lifecycle-aware architecture when you are writing your Android application. It is not specific to coroutines, but applies to any Android application that performs any kind of long/asynchronous operations. The official Google’s guide to app architecture is a good read that explains the problem and walks though a solution: Guide to app architecture  |  Android Developers

TL;DR: you code that interacts with TTS engine should be in your view models and should not be directly tied to the lifecycle of your activities. TextToSpeechManager should not keep a reference to a context.

You will indeed need to use a future to start loading your TTS engine as soon as possible. However, in your original code you don’t have to do suspendCoroutine inside async, you can simply use CompletableDeferred:

private fun init(): Deferred<TextToSpeech> {
    Log.d(tag,"init")
    return CompletableDeferred<TextToSpeech>().also { fut -> 
        selfReference<TextToSpeech> {
                // replace con.resume with fut.complete in this code
        }
    }
}

As a matter of style, I would recommend to rename init to initAsync (because it returns a future)

Now, notice that get implementation in your original code is racy. It first checks isCompletedExceptionally and then invokes await. A classical “check-and-act” problem. The initialization might complete exceptionally between those checks and get will throw an exception instead of retrying to initialize and it was designed to. To fix it rewrite:

private suspend tailrec fun get(): TextToSpeech {
    Log.d(tag,"get")
    try { return ttsDeferred.await() }
    catch(e: Throwable) { 
        // todo: log init error here
        ttsDeferred = initAsync() // start init again
        return get() // tailrec call to retry
    }  
}
2 Likes
5

Thank you!
CompletableDeferred is what I was looking for - combination of async and suspendCoroutine. It is similar to classical deferred object in web and Deferred class I used in my project.

Yesterday I found the reason of leakage. If I resume continuation with line flushed.continuation.resumeWithException(Exception("UTTERANCE_FLUSH")) I get strange exceptions or silent app-termination, but not UTTERANCE_FLUSH message in logcat. No ANY error messages in logcat!
I replaced my click listener with

button.setOnClickListener({
    launch {
        try {
            Log.d("APP", "speak start")
            ttsMan.speak(text.text.toString(), TextToSpeech.QUEUE_FLUSH)
            Log.d("APP", "speak end")
        } catch (e: Throwable) {
            Log.d("APP", "speak error", e)
        }
    }
})

and saw expected message UTTERANCE_FLUSH and no leakage, strange exceptions or silent termination!

I think this is a lack of exception handling in Kotlin coroutines, because I could spend a lot of time to get the crash reason if I forget to wrap suspend function with try/catch.

In java-world I used Checked Exceptions for it. I always knew when it might be some kind of shit.
As I understand, Kotlin doesn’t support this way by design.

The second variant - suppress any uncaught errors in coroutines. It is the usual way of javascript es6 promises:

Promise.reject("sample error").then(value => console.log(value)); // prints "Uncaught (in promise) sample error", but no termination!

The third variant - throw the original exception.

Is it possible to support one of these variants now or in the future?

6

The upcoming version of kotlinx.coroutines will log uncaught exception on Android. See this issue: Coroutines do not log uncaught exceptions in Android · Issue #148 · Kotlin/kotlinx.coroutines · GitHub

We cannot ignore uncaught exceptions by default. So, when you launch a coroutine any uncaught exception is going to crash your app (with log in the upcoming release).

However, you always have an option to handle exceptions. There are many way to do it. However, using raw try { ... } catch ... in your application code is not idiomatic in Kotlin. Define your own higher order function:

inline fun catchAllLog(block: () -> Unit) = 
    try { block() } 
    catch (e: Throwable) {
        Log.d("APP", "speak error", e)
    }

and use it when you know you need to ignore & log exceptions: catchAllLog { ... }.

Alternatively, with coroutines, you can define your own handler for uncaught exceptions:

val catchAllLog = CoroutineExceptionHandler { e -> 
    Log.d("APP", "speak error", e)
}

then use it when you launch your coroutines: launch(catchAllLog) { ... }.