How to fetch paged resource and accumulate results in a functional style?


#1

I’m currently writing some code to repeatedly call an endpoint that returns a recordset and paging token, which indicates whether there is further data to fetch. Pretty common situation, I imagine, and here’s what I’m using to do this (using Amazon’s AWS SDK):

fun ECSClient.fetchAllServices(request: ListServicesRequest): List<String> {
    val services = mutableListOf<String>()
    var page: ListServicesResponse
    var tokenizedRequest = request

    do {
        page = listServices(tokenizedRequest)
        services.addAll(page.serviceArns())
        tokenizedRequest = request.toBuilder().nextToken(page.nextToken()).build()
    } while (page.nextToken()?.isNotBlank() == true)
    return services
}

This is a perfectly fine imperative implementation, but having become enamored of functional style recently, I find myself wondering if there’s a way to accomplish this with a Java streams-esque series of functional calls, and I haven’t had any success finding such a thing with a Google search.

The closest I can imagine is maybe creating an extension function that acts as a generator, making the listServices() call within it and yielding the results, and evaluating the page token to determine whether to terminate generation. I’m a little fuzzy on how that would be implemented, though.


#2

Hi @mrtact
I worked on the same task but unfortunately I cannot send you my solution now.

However I replaced the do ... while block with http://kotlinlang.org/api/latest/jvm/stdlib/kotlin.sequences/generate-sequence.html and then map/flatten the results.
So your function returns a Sequence you can load the data lazyly and you don’t have to put all in memory (or use toList if you need it).

Otherwise you can generate a sequence putting your code in https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.coroutines.experimental/build-sequence.html