Proper way of networking and file operations using Coroutines

Support
1

Hi community
I’m looking for the best practice for a specific network and file operation using Kotlin coroutines. In this particular case, I cannot use Retrofit2 which supports coroutines and suspend functions, because I need some special cancellation of download process and publishing progress. So I decided to use Okhttp3 an Okio libraries. As far as I realized, just putting the blocking networking and file-writing functions inside another suspend function doesn’t make sense, because they still block the thread. Here’s my current(wrong) code:

suspend fun download(url: String, destFile: File, downloadId: Int) {
    val request = Request.Builder().url(url).build()
    val response = OkHttpClient().newCall(request).execute()
    val body = response.body
    val contentLength: Long? = body?.contentLength()
    val source = body?.source()
    
    val sink = destFile.sink().buffer()
    val sinkBuffer = sink.buffer

    var totalBytesRead: Long = 0
    val bufferSize: Long = 8 * 1024

    var bytesRead: Long?
    while (true) {
        bytesRead = source?.read(sinkBuffer, bufferSize)

        if (bytesRead == -1L)
            break

        sink.emit()
        totalBytesRead += bytesRead!!

        println((totalBytesRead * 100) / contentLength!!)
    }

    sink.flush()
    sink.close()
    source?.close()
    
    println("DOWNLOAD FILE $contentLength DONE")
}

I found this library which is an await extension for Okhttp3 and is used like this as a suspend function:

val result = client.newCall(request).await()

But there’s no such thing for Okio file operations and since I need the progress and cancellation ability, I have to use that while loop there.

Now here’s the actual question : Is it fine just to wrap the while loop in withContext(Dispathers.IO) {...} ? Or it’s the same mistake as the beginning which blocked threads and resources? And if it’s fine why to use that extension for actual requests instead of wrapping them in the same withContext block?

2

Disclaimer: I’m not 100% confident about my explanation below. If anyone spot any mistakes or inaccuracies, please let me know.

Yes, we should not use blocking IO inside suspend functions directly. And yes, putting such operations inside withContext(Dispatchers.IO) {...} is a common and proper way of handling them. It doesn’t magically make threads to not block, it just delegates a blocking operation to a dedicated pool of threads that are meant and optimized for blocking.

This is actually a good question and it is related to blocking vs non-blocking I/O. As said above, using Dispatchers.IO still blocks threads. We use it to avoid blocking coroutines, so to keep our application responsive and utilize CPU fully, but threads are still blocked and that means we waste some resources, mostly the memory.

Some network libraries support non-blocking I/O and that means we can use fewer number of threads than if using blocking I/O. To accomplish this they usually provide a callback-based API. With coroutines we can utilize their non-blocking I/O while still writing sequential code, using suspend functions. This is exactly what this await() extension do.

I believe Okio does not support non-blocking I/O (correct me if I’m wrong) and that means making suspending API makes little sense for it. You can just use Dispatchers.IO.

Also, you said you need the progress capability. This may be harder with non-blocking I/O, because the networking framework has to explicitly support such feature with its callback API. This is much easier done with traditional blocking I/O.

Conclusion: I would go with your original code, just put it inside Dispatchers.IO.

3

Great appreciation for your clarification and such responsibly-written answer.

Yes you’re right.

Yes it is, but I want to use the highest performance code, because downloading operation is supposed to be applied to a list of multi-media items and improper way of handling it would result into performance failures.

As I realized from your answer, generally putting blocking functions without non-blocking support into a withContext(Dispatchers.IO) {...} and then taking it as suspend function(like it had non-blocking support from the beginning) is a best-practice? Since there will probably be a lot of such operations in every project.

4

I suggest running some benchmarks for blocking vs non-blocking. I know common understanding is that non-blocking is better for performance, but I think this is oversimplification. It is definitely better for the memory, but I wouldn’t be so sure about the CPU. But again, run some benchmarks :slight_smile:

Yes, this is a very common practice and is similar to what many networking libraries internally do (they manage an I/O thread pool).

If you plan to perform a really heavy IO, it could be a better idea to not use Dispatchers.IO, but create your own dispatcher. Dispatchers.IO is by default limited to 64 threads (I believe) and it is shared across the whole application, so using it by your heavy components could impact responsiveness of a lighter, but more critical component.

5

You’ll also want to check in your while loop if the coroutine has been cancelled and abort cleanly.

6

Thank you for your response
What do you exactly mean by “abort cleanly”? I know about isActive and ensureActive().
Let’s say they’re false. What should I do?

7

Stop writing the file and do any file cleanup needed.