Kotlin nullable types: dealing with ambiguous results?

Language Design
1

For fun, I was looking to port a pile of Java code I’d written with Optionals over to Kotlin and I ran into an interesting issue that I can illustrate with this function from the Kotlin stdlib:

/**
 * Returns the first element, or `null` if the sequence is empty.
 *
 * The operation is _terminal_.
 */
public fun <T> Sequence<T>.firstOrNull(): T? {
    val iterator = iterator()
    if (!iterator.hasNext())
        return null
    return iterator.next()
}

Consider if we’re dealing with a sequence of a non-nullable type, like say Sequence<String>. At this point, it’s pretty clear what happens. If the sequence is empty, you get null, otherwise you get the String you’re looking for. But what happens if the concrete type is Sequence<String?>, so nulls can exist in the sequence? Now you’ve got a more interesting problem, because you will get null in two conditions: if the head of the list is null, or if the list has nothing in it.

In Java8, you might use Optional<String> for the return type, but Optional.of(null) throws an exception. It requires its input to be non-null. Because of this, oddly enough, the standard idiom in Kotlin is somewhat more ambiguous than the verbose way Java8 wants to make us solve the same problem.

Why does any of this matter? If you’re writing another generic function over Sequence<T> and you just assume that calling firstOrNull() will only return null when the list is empty, your code will be just fine for non-null sequences, but will have a hidden logic bug because you didn’t think about nulls.

Example code to demonstrate:

private fun <T> Sequence<T>.status(): String = firstOrNull()?.toString() ?: "nothing there!" 
private fun <T: Any> Sequence<T>.status2(): String = firstOrNull()?.toString() ?: "nothing there!"

@Test
fun testSequenceNullability() {
    val emptyList: Sequence<String> = sequenceOf()
    val nonNullList: Sequence<String> = sequenceOf("Alice", "Bob", "Charlie")
    val nullsIncluded: Sequence<String?> = sequenceOf(null, "Alice", "Bob")

    assertEquals("nothing there!", emptyList.status())
    assertEquals("Alice", nonNullList.status())
    assertEquals("nothing there!", nullsIncluded.status())
    assertEquals("nothing there!", nullsIncluded.status2()) // compiler error
}

Here, status() is the “obvious” implementation, while status2() refuses to run on sequences with nulls in them, which ensures that a return value of null from status2() always implies that the sequence has no entries. No more ambiguity.

Food for thought: should Kotlin stdlib functions like firstOrNull() that return nullable values be written more like my status2() example? Or, does Kotlin need an Option/Maybe/Result type of some sort, such that a function like firstOrNull() has a way of distinguishing these cases in its output?

2

Probably not what you’re looking for, but I feel that collections of nullable objects are generally bad style. You could use something like the null object pattern instead, which would solve the problem.

Also what’s wrong with calling isEmpty and first separately? I know in kotlin you can put a lot of stuff into one line, but that doesn’t mean you always should.

Often, slightly more verbose code is easier to read.

3

The code I was looking to port from Java8 was managing lists of Optional<T>, which are easy to get when you’re mapping a function on a list that returns an optional.

The idiomatic equivalent in Kotlin are sequences of T?, so it’s important to see how all the various stdlib methods operate on these sorts of things. My tentative conclusion is that stdlib methods on Sequence<T> need to be careful about their assumptions, and many need to be restricted to only operate on non-null types, ie, <T: Any>.

4

You shouldn’t really do that according to most guidelines, either. Optional is meant mostly as a return value for some functional methods like Stream.first().

What? I find nothing wrong with List<T?>.firstOrNull() returning null both on emptiness or on the first element being null. That’s exactly what the method says. Why artificially restrict it?

5

It’s a standard practice in the world of functional programming to use an Optional/Maybe/Result type as a way of indicating whether a function has an internal error or whether it could successfully do whatever it was meant to do. Kotlin rolls this style of programming into its safe handling of null results, forcing the caller of such a function to deal with failure cases by testing for nullity. If you don’t check, then your program doesn’t compile.

If you have a list of whatevers and you map such a function onto the list, you’ll get back a list of optionals (or, in Kotlin, a list of possible nulls). At that point, the natural thing to do is to filter the list to remove the Optional.empty values and then map again to get only a list of the Optional.of values.

Some functional libraries treat an Optional/Maybe/Result type as yet another sort of “sequence”, in that an Optional looks an awful lot like a list with zero or one entries in it, meaning you can go through the above filter and map pipeline in one go by using some sort of flatmap call on the original list. Kotlin does this with filterNotNull.

So now let’s get back to the function at the top of my original posting. If the concrete type T is something non-null, then the documentation is exactly correct. You’ll get back null if the sequence is empty, otherwise you’ll get back the head of the sequence. All is right and proper with the world. But this particular function makes no nullity constraints on the sequence. If the concrete type can be null, which is allowed here, then the documentation is wrong, in that getting back null does not actually tell you whether the sequence was empty. It only implies that it might have been empty.

Were somebody to implement code that used firstOrNull, they might be counting on this behavior (i.e., null implies empty), which could yield unexpected bugs, which is pretty much the whole point of why coding with nulls, in Java, is like playing with fire and why we have all these recent Java libraries that forbid nulls, and why we have annotations that can enforce these properties.

So back to Kotlin: were firstOrNull to make a type constraint, <T: Any>, it would then be unambiguous again and the documentation would accurately describe its behavior. But the code, as written, disagrees with its documentation. This is exactly the kind of ambiguity that Kotlin’s nullity or non-nullity type annotations are meant to resolve.

If the code and the comments disagree, then both are probably wrong.
– attributed to Norm Schryer

6

If you get rid of the Optionals as soon as possible (which is how Optional is intended to be used), you won’t have this problem. Pseudocode (which will probably have warnings about getting a value without checking that it is present):

val listOfPossiblyEmptyResults = ...
val listOfResults = listOfPossiblyEmptyResults
         .filter { it.isNotEmpty() }
         .map { it.get() }
7

My typical reaction to questions of this sort is to question the requirements.
“Why is Optional used in a list?”.
Likely Answers

  • Its an artifact - should be ingored.
    A: dont put them in the first place, or remote them immediately similar to dwallach’s suggestion.

  • They are important, they ‘mean something’.
    A: Implement Optional - its trivial, and you can find the Java source in openjdk.
    Q: WHY are they important – WHAT do they mean ?
    A: where possible iterate up to as close to the actual business requirements.
    At each step repeat the same questions.

  • Ultimately you will find that NO data structures are actually required explicitly, they are all artifacts of arbitrary encoding of business requirements into software abstractions, the language or framework used to implement it are not the ‘best’ ones in your opinion, There are vast areas of ill-conceived or misunderstood implementations and designs – or so it seems, they may simply be written by a mad genius intent on causing as much confusion as possible; basically the fact it runs at all is the biggest mystery. Ending in a disturbing thought that maybe its only you who doesn’t see the divine heavenly inspired elegance of a masterpiece cleverly disguised as a pile of FUD as to be recognized only by the pure of mind worthy of judgment and respect.

The Kings New Clothes.

Not that helpful in itself.

The side effects of the process - in any scale - are very helpful.

In the end this is simply the process of understanding the entire system (human and computer), to the extent
necessary to rewrite it from scratch. – OR to determine what is most expedient and beneficial to rewrite vs adapt both in the specific case and as a strategy for the whole project.
If that is too large a scope then you ‘wing it’ and move on. You wont get it ‘absolutely right’ (there is no single ‘right’) but you can get it to work, and perhaps ‘right’ for some definition of ‘right’. (there are many).

Buy or Build. The kind of decision making that people are better at then computers (today).
Which is why the compiler and libraries are NOT complete or consistent in all cases
The primitives to achieve any result are available and easy, but since this is an integration issue, not a ‘pure language’ issue – there is no ‘right’ choice – for either you or the compiler authors – therefore which choice to make is left to you.