Feature proposal: lambda flattening syntax to enable RAII and more elegant code

The Problem

In the context of multi-threaded programming, or dealing with a lot of objects with a “lifetime,” we often are forced to write code like this:

lockA.read {
    lockB.write {
        ...
    }
}

use(ResourceA()) { resourceA ->
    use(ResourceB()) { resourceB ->
        use(ResourceC()) { resourceC ->
            ...
        }
    }
}

Meanwhile in C++, programmers often simplify such places using RAII, or in other words, using stack objects’ constructors and destructors to handle scope or lifecycle automatically:

void MyFunction() {
    ReadLock lock(&someLock);
    ... // The lock will be released when the function is complete.
}
void MyFunction2() {
    ReadLock lockA(&someLockA);
    ReadLock lockB(&someLockB);
    ... // The locks will be released in reverse order of declaration.
}

Setting aside the verbosity of C++, this syntax is much more concise, and accustomed programmers will have no difficulty reasoning about its meaning.

The Solution

While we don’t have stack-based object allocation in Kotlin, we could still reduce the number of lines and indentation with a purely syntactic change. The proposal is simply this: allow us to invoke lambda-accepting functions and pass their variables into the current scope without extra indentation or curly braces! Here’s the proposed syntax:

val resource = &use(MyResource())
...

val (x, y, z) = &myFunction()
...

&with(MyContext())
...

These calls would treat all code between them and the current scope’s closing brace as though it were inside a new lambda passed into the function – in other words, the following is equivalent to the above:

use(MyResource()) { resource ->
    ...

    myFunction { x, y, z ->
        ...

        with(MyContext()) {
            ...
        }
    }
}

Functions callable in this manner would be restricted to those annotated with contract { callsInPlace(block, InvocationKind.EXACTLY_ONCE) }, since other cases necessarily involve transforming code in a bizarre way that falls outside helpful use cases, in my opinion.

With these changes, the examples given at the start would look like this:

&lockA.read()
&lockB.write()
...

val resourceA = &use(ResourceA())
val resourceB = &use(ResourceB())
val resourceC = &use(ResourceC())
...

Caveats

The contract pattern given above should be mandatory in my opinion. However, since the point of contracts is to alleviate the work of validation from the compiler by trusting the programmer, this feature could still be abused to create some truly bizarre code:

inline fun <T> Array<out T>.forEach2(block: (T) -> Unit) {
    contract { callsInPlace(block, InvocationKind.EXACTLY_ONCE) } // WRONG
    this.forEach(block)
}

fun myBizarreCode() {
    val array = arrayOf("x", "y", "z")
    val v = &array.forEach2
    println(v) // gets invoked three times!
}

However, this would be a pretty extreme case in my opinion – even the most basic language features can be horribly abused, but we still allow them to empower the best programmers, not protect against the worst. That aside, I would argue the above case falls well outside of ignorance and into malicious (if not hilarious) behavior.

Another possibility is simply not requiring a contract and always allowing the code transformation - while this could perhaps empower DSLs in some way (which are already fairly abstract as far as code goes), I think it’s too easy to misuse for a newcomer. The existing callsInPlace contract pattern maps very well onto use cases we’d ever like to support, in my opinion.

Aside from requiring a certain contract pattern, the biggest issue is discarding the called function’s return value if non-Unit. I don’t believe restricting this syntax to functions returning Unit is advisable, since we would often like to call functions returning a generic type, which often resolve to non-Unit despite not being used as such. If the return value is needed for any reason, then it should simply be that you cannot use this syntax, as the extra curly braces and indentation are necessary to delineate lambda from non-lambda code in that case.

EDIT: another concern is how to handle when the lambda has a return value. The Unit case is naturally fine, and the generic case can always infer a type of Unit. For other cases, I’m not sure there’s a solution more permissive than simply disallowing it. Trying to handle the return type in some intelligent way would likely complicate semantics around return/break/continue, and I don’t see a use case for it yet.

Alternative Syntax

I’m not at all decided on the specifics of the syntax. I thought it might be neat to reuse the arrow syntax somehow, but this seemed less clear to me than using the ampersand:

val resource <- use(MyResource())
val resource -> use(MyResource())
val resource => use(MyResource()) // I like this one, but C#'s existence makes this unclear imo
val resource = ^use(MyResource()) // very unique symbol, but bad parallel to managed GC in C++
val resource = *use(MyResource()) // parallel with spread operator, and dereference in C++
val resource = &use(MyResource()) // yet unused symbol, and parallel to reference in C++

I like the ampersand because it’s not used anywhere else in the language to my knowledge, and has parallels to C++. The star was a close contender to me, and even better in terms of analogies, but its smallness and use elsewhere make it seem less clear.

The use of val and flipping the variables to the left-hand-side helps explain the feature imo - the arguments are very obviously introduced into the current scope as local variables, which is a major understandability win.

The weakest syntactic case imo is here:

&lockA.read() // simply the val syntax without the left-hand-side
lockA.&read() // symbol closer to function of interest, but harder to notice

At least having the ampersand on the very far left lets the reader instantly know something special is happening on this line. In your head, you know “when I see an ampersand, some closing code will be invoked later.”

5 Likes

Interesting idea. You considered some good cons as well.

My initial take is that I’d be curious to see how this compares to the alternative of leaning into the “context-based programming paradigm”, especially since that seems to be the way Kotlin is going with Decorators*, Contexts, Namespaces, etc.

Specifically, I wonder if either 1) there might be an existing alternative or an alternative combined with Contexts, that fills this proposal. Or 2) if this proposal might choose a syntax/system that aligns with Contexts in some way?


Here’s my initial thoughts for #1:
I think resource objects are a decent example of the problem of “Lifetime” objects. Right now I’d say the common way of solving this is to provide the scope and work in that context. So the first example would become:

resourceScope { // Notice that we keep the same indentation
    val resourceA = use(ResourceA)
    val resourceB = use(ResourceB)
    val resourceC = use(ResourceC)
}

The benefits here are that its idiomatic Kotlin, it’s code we control, and it’s a library solution instead of a language one.
If we don’t like the context methods of use(myResource) we can change it to anything else. Maybe an extension function myResource.using() or anything else we want.
Maybe we want to support different kinds of ResourceScopes that behave differently or pool resources, handle closing resources in a special way, or even allow users to integrate their own implementations to our library–all of it is supported out of the box in normal Kotlin.

The cons to this currently available option have been that Kotlin hasn’t had multiple-receivers. Soon it will with Contexts.
This is the same con that comes with coroutines and having to extend CoroutineScope for your functions that launch new coroutines. It takes up your one and only receiver for the function.


For #2,

It’s true Kotlin doesn’t have a “stack” scope that you can extend or act on (like a decorator). Instead of solving the issue of writing, lambda’s via this proposal, why not ask for a way to hook into the current scope/stack within a method? Maybe an extension function format like fun scope.use(r: Resource) or context fun foo() that lets you decorate or hook into the enclosing scope? Personally I’m not too much of a fan of this one but I do think it’s closer to a Kotlin-style solution for Stack/Scope aware functions.


TL:DR

I’d respond with: We do have context-aware programming in Kotlin, which enables scoping for lifetimes/lifecycles.
Right now it does come at a cost, which is one of the main reasons “multiple-receivers” has been of interest for so long.
Framing this proposal in light of the upcoming Contexts would help evaluate what improvements it brings.

2 Likes

Thanks for the reply, these are really great points.


For #1, I would say it’s definitely a question of tradeoffs. You can absolutely do this in a context-oriented way, however it comes at the following costs:

  1. Having to implement said functions, or ship them with the standard library, when existing functions can be repurposed for this at “no cost,” because they have the same semantic meaning.
  2. One extra layer of indentation (at the risk of sounding pedantic.)
  3. Cost of bookkeeping – taking your method as an example, we would have to allocate a List behind the scenes so that we could traverse the resources at the end and destroy them. While in many situations this is perfectly acceptable, it’s often not for “performance critical” code, i.e. mobile, game, embedded, finance, or scientific computing, where we’d really like to avoid invoking the GC or polluting the cache to the greatest extent possible. (This goes doubly so for multi-threaded code.) Considering that Kotlin already goes to great lengths to conserve performance by not boxing primitives on the JVM (for example), I believe this is a valid concern for the language.

For #2, I’m more interested in alleviating the clutter of many lambdas in one place than I am in supporting full stack-based semantics. (You can actually emulate a lot of this as-is by using ThreadLocals of objects with inline methods, with acceptable performance for real-time use cases. The problem is its a lot of implementation complexity with room for error, and not quite as fast as a true “stack,” especially because nothing is getting saved to registers.)

Nested lambdas are inherently a “stack-like” pattern, however, and while the new context features can reduce this by offering an alternate implementation, they’re not going to remove lambdas from the language. So why not make lambdas more elegant, capitalizing on what we already have? (Also, I wouldn’t expect the standard library to offer a reimplementation of every relevant lambda function in order to accommodate a context-based usage pattern.)

In terms of aligning with the new context features, that was actually the initial motivation for this post - some users were debating a new with val MyContext() feature to inject contexts into the current scope, and I thought this proposal would cover that (e.g. &with(MyContext())) while improving the language more generally, satisfying some other use cases I’ve been running into. I suppose introducing contexts alongside an acquired resource could also be useful:

val resource = &acquireResource() // also introduces a context
resource.doAThing() // context extension method (maybe we don't own resource's class)

I think it would be worthwhile to consider every standard library function this could be used with – once I have some time I’ll try and do that.

1 Like

Interesting. I’ll have to refresh on the Contexts KEEP–it’s been a while.

You also mention performance hits due to allowing arbitrary code to decorate the lambda. I disagree that it would be an issue. I’d say your point (2), nested lambdas is the strongest supporter of the proposal.

I wonder if there’s another syntax that would match Context Receivers more? I guess I’d rephrase this proposal as, entering the context of a trailing closure argument without a layer of nesting.

If I understand correctly, while Context Receivers support doing

with val transaction = Transaction() // Introduces the context but not in a closure
insertRow()

But they don’t support doing

with val myList.forEach() // Introduces the context but ALSO a closure
println(it)

^ Not that the “with” or “val” make much sense but it helped me see the difference

I suppose my grievance is with the following:

with val transaction = Transaction()
insertRow()

Contexts are introducing a solution to a problem that isn’t exclusive to contexts! (I.e. excessive nesting.) So why are we approaching it like a context-specific issue, when the language could more generally benefit?

(EDIT: btw the forEach case would be disallowed under both the context KEEP and what I’m proposing here, though only through the power of contracts.)

2 Likes

Finally some good language design proposals haha

2 Likes

Right gotcha. Reading through the the notes in code coloring (really awesome stuff btw) makes me wonder if it will get solved with upcoming changes.

For example decorators in this example get ride of the nested transaction.use { }.

// Declaring transactional decorator
decorator fun transactional(block: Transaction.() -> Unit) {
    beginTransation.use { tx ->
        tx.block()
    } // Closes transaction at the end    
}

@transactional // Use decorator to wrap function's body
fun updateUserSession() {
    val session = loadSession()
    session.lastAccess = now()
    storeSession(session)
}

Technically we aren’t changing the closure within the same block of code, but otherwise it seems like it would work–depending on how decorators end up working.

If this needed to be done within a code block and not the entire function, I imagine you might be able to compose decorators together with a single level of nesting :crossed_fingers:

This still isn’t exactly the same but I wonder much of these use-cases composing decorators and providing a single lambda would solve. And maybe this proposal / use-cases should be considered whenever decorators get their design fleshed out.

EDIT: Just wanted to note that the chained lambda syntax sugar does not do everything this proposal does. Same with decorators. Just trying to put the right pressure on it since it’s probably worth a Gist if examples and comparisons

1 Like

In my opinion, @arocnies’ solution with resourceGroup introducing a context receiver is idiomatic Kotlin and solves the problem efficiently without requiring any new langage feature.

I’m personally worried about introducing any feature facilitating RAII-style patterns to the language. Kotlin doesn’t have lifetimes for objects, only for variable bindings. Kotlin also doesn’t have a concept of destructor.

val foo = MyResource()
run {
  val bar = &use(foo)
  // do stuff
}
println(foo)

I think that objects that can be ‘closed’ are an anti-pattern in Kotlin, because the language just cannot protect the user from using them. Of course they are sometimes necessary, but I don’t think they should be encouraged. For that reason, I think the standard library’s use is enough.

With the implementation as written, yes, but for performance critical code it is easy to declare a resourceGroup(resource1, resource2) { … } etc variants with a specialized implementation.

The main difference is that the with keyword has no behavior whatsoever, so it’s not an issue that it does “magic”. It’s important that the language remains explicit, where it’s obvious what does or doesn’t run what code. with … context has no observable behavior other than introducing a variable in scope, which would otherwise be there anyway as a regular explicit variable.

1 Like

I like this idea generally, but I think it’s better to just omit the braces

use(MyResource()) resourceA ->
use(MyResource()) resourceB ->
1 Like

Good points. And this is a great explanation of how this proposal differs from with context.

I think a single flattened closure with brackets would be the balanced counter if something were to be added. Having at least one closure avoids throwing in magic into all the following lines and hiding the fact that they cannot be considered equal with the code above the entry to the closure.

Maybe something closer to the chained lambda calls make more sense if it’s important to keep the distinction of code that’s outside vs inside the closures.

use(MyResource()) -> transaction() -> withTimeout(5000) -> {
    // code
}
// Or typed on multiple lines
use(MyResource()) ->  
withTimeout(5000) -> 
transaction() -> {
    // code
}

^ Not that I’m super keen on that yet either. I can spot a few issues with return types–just brainstorming.

The downside is that this drops the ability of the OP proposal to run code between the layers. I wonder if that’s not as important depending on the IRL use cases? It might be worth pulling up the chained lambda discussions and seeing what ideas those landed on.

Still, I suspect contexts will have a larger impact on these use cases and I’m curious to see what options people come up.

EDIT: Here’s a link to one discussion on simplifying nested lambda’s: More concise nested lambdas - #2 by broot

1 Like

If Kotlin were to add some python-like *args feature but in a typed matter, then maybe something like this (imaginary syntax with params) could do the trick:

inline operator fun <params F1Params, params F2Params, params F1Contexts, params F2ContextsFromOutside, params F2ContextsFromInside,
        F1R, F2R, F2LambdaR, params F1LambdaParams, params F2LambdaParams, params F1LambdaContexts, params F2LambdaContexts>
(context(F1Contexts) (F1Params, context(F1LambdaContexts, F2ContextsFromInside) (F1LambdaParams) -> F2R) -> F1R)
        .times(crossinline f2:
               context(F2ContextsFromOutside, F2ContextsFromInside)
                   (F2Params, context(F2LambdaContexts) (F2LambdaParams) -> F2LambdaR)  -> F2R):
        context(F1Contexts, F2ContextsFromOutside)
            (F1Params, F2Params, context(F1LambdaContexts, F2ContextsFromInside, F2LambdaContexts) (F1LambdaParams, F2LambdaParams) -> F2LambdaR) -> F1R {
    return { f1Params, f2Params, function ->
        this@times(this@F1Contexts, f1Params) {f1LambdaParams ->
            f2(this@F2ContextsFromOutside, this@F2ContextsFromInside, f2Params) {f2LambdaParams ->
                function(this@F1LambdaContexts, this@F2ContextsFromInside, this@F2LambdaContexts, f1LambdaParams, f2LambdaParams)
            }
        }
    }
}

used like this:

(::foo * ::bar)(...) {
  // code goes here
}

Hmm, on second thought, I can see why such a feature isn’t included, which is because it’s an absolute spaghetti mess. Just food for thought, though, because decorators promises to at least wrap functions while only providing contexts, not arguments, while this mess also adds arguments and return types, and basically allows both functions full-control over how the code is executed

1 Like

You can easily create a stack based solution:

import java.io.Closeable
import java.util.*

interface ResourceContext {
    fun <T : Closeable> use(obj: T): T
}

fun resourceBlock(block: ResourceContext.() -> Any) {
    val context = ResourceContextImpl()
    block(context)
    context.freeAll()
}

private class ResourceContextImpl(private val resourceStack: Deque<Closeable> = LinkedList()) : ResourceContext {
    override fun <T : Closeable> use(obj: T): T {
        resourceStack.push(obj)
        println("$obj was added")
        return obj
    }

    fun freeAll() {
        while (resourceStack.isNotEmpty()) {
            val obj = resourceStack.pop()
            println("$obj was closed")
            obj.close()
        }
    }
}

usage:

resourceBlock {
    val str1 = use(StringReader("baba"))
    val str2 = use(StringBufferInputStream("yaga"))
    str1.read()
    str2.readAllBytes()
}

Output:

java.io.StringReader@64b8f8f4 was opened
java.io.StringBufferInputStream@3cd1f1c8 was opened
java.io.StringBufferInputStream@3cd1f1c8 was closed
java.io.StringReader@64b8f8f4 was closed

I used a Closeable interface but of course it can be changed to whatever you use.

[EDIT] Ooopsie, I can see that arocnies proposed that in the very first response. Sorry for my clutter! I’ll keep the post anyway in case somebody finds it’s useful.

3 Likes

As a C# programmer that has recently turned to Kotlin, one of the most frustrating moments in my learning journey was discovering that the auto closing resources mechanism required adding a level of indentation for each resource, wether in C# it has never required more than a single level for all resources initialized together and, with recent improvements, does not require indentation at all. I believe it is extremely positive that this and other similar concerns (locks for instance) are addressed, and I believe that it would blend very well with the multiple context receivers that is in development.

I also would like to add that, in my opinion, closeable objects are not an antipattern but rather a necessity.

2 Likes

You can have multiple resources with one level overall if you define your own extension like so:

inline fun <A: Closeable, B: Closeable, C: Closeable, D: Closeable, E: Closeable, F: Closeable, G: Closeable> using(a: A, b: B, c:C, d:D, e:E, f:F, g:G, block: (A, B, C, D, E, F, G) -> Unit) {
    block()
    a.close()
    // and so on
}
// or if you'd like to use them as context receivers:

inline fun <A: Closeable, B: Closeable, C: Closeable, D: Closeable, E: Closeable, F: Closeable, G: Closeable> using(a: A, b: B, c:C, d:D, e:E, f:F, g:G, block: context(A, B, C, D, E, F, G) () → Unit) {
block()
a.close()
// and so on
}