Null-safety and Java reflection


#1

Hi,

I am working on a Spring Boot project and came across the situation that nulls get set onto non-nullable variables by Spring.
Since Spring is just using Java reflection the simplified version is the following:

import java.lang.reflect.Field

class MapSB(var map: MutableMap<String, Boolean> = HashMap())

fun main(args: Array<String>) {
    val foo = MapSB()
    // simulation of spring magic happening ------
    val nullMap = HashMap<String, Boolean?>()
    nullMap.put("one", null)
    nullMap.put("two", null)
    nullMap.put("three", true)
    val field: Field = MapSB::class.java
    		.getDeclaredField("map")
    field.isAccessible = true
    field.set(foo, nullMap)
    // -------------------------------------------
    // NPE 1
    for ((k, v ) in foo.map) { // <- here
        println("$k,$v")
    }
    // NPE 2
    for (entry in foo.map) {
        val k = entry.key
        val v = entry.value ?: false // <- here
        println("$k,$v")
    }
    // Warning condition 'entry.value != null' always 'true'
    for (entry in foo.map) {
        val k = entry.key
        val v = if (entry.value != null) entry.value else false
        println("$k,$v")
    }
}

Currently I am just suppressing the warning but I wonder if there is a better way to deal with the problem.
I don’t want to change the declaration to var map: MutableMap<String, Boolean?> because then I would have to deal with null-checking in all code using the MapSB class.


#2

You need Map implementation what disallows nulls as values. The simplest example:

class NotNullMap<K, V>(private val map: MutableMap<K, V>) : MutableMap<K, V> by map
{
    override fun put(key: K, value: V): V? {
        if (value == null) error("null values not allowed")
        return map.put(key, value)
    }
}

But probably need to override several other mutation methods.

Perhaps it’s implemented already in some libraries, but not sure. Maybe look at Guava, Apache Commons Collections or somewhere else.


#3

Thanks for your input but after re-reading about Kotlins null safety var map: MutableMap<String, Boolean?> is the recommended way to deal with this which is a little disappointing.


#4

The question really is not about Kotlin itself, but about Java interoperability. Spring works on Java side and follows Java rules. If you want prevent nulls in Java, you will need some additional code.


#5

Well actually it is about Kotlins Java interoperability and therefor about Kotlin itself in my book.
Or is Java interoperability not part of Kotlin?
What are you trying to say here?

Sure, but the example code does not use Spring, does it?

I never said I wanted that. It was your idea.
What I wanted is no incorrect warnings, broken elvis operators and of course no null values in Kotlin variables of non-nullable types.
And this I get by using nullable types and doing my null checks in Kotlin as told by the documentation.

for every type you use to interact with Java.
There, completed that for you. Because it’s not about maps. Replace the Map with an Array<NonNullableType>, Integer, String, etc.
Are you recommending to implement non-nullable Integers and Strings in Java?
The point is that null-safety is gone the moment you interact with Java (even type-safety is gone when using Java reflection as I can happily assign i.e. a String? to a String).
Using nullable types in Kotlin is the right thing to do. Hiding that behind “non nullable Java types” (when there isn’t even the concept of not nullable in Java) is neither necessary nor good design IMHO.


#6

What you’re saying is, there are no guarantees once you interact with code you have not written yourself? Yes you can use reflection to do some unsafe stuff. The problem is that the jvm, as you pointed out, knows nothing about nullable and not-null values. Once you interact with code at that level everything is nullable. Also you are using generics, so at runtime there is not even a difference between List<String> and List<Foo> due to the type erasure.
The question is how the language handles edge cases like this. Kotlin decided to build a system which is powerful and easy to use and is interoperable with Java. Are there edge cases, which can lead to NPEs? Yes, your example is one, another is accessing fields before the constructor is finished.
The problem is that kotlin is restricted by the JVM, which means that using java reflection there are always ways to set fields to null which should not be. But that is precisely what NPEs are for. They get thrown if you do something unsafe and it fails.
Kotlin never claimed that NPEs will never occur. It only claims that it will help prevent them when possible. No language can prevent all bugs, it can just help to reduce the number of bugs. In the end it is still up to the programmer.


#7

As shown by incorrect warnings, broken elvis operators and NPEs.
I already got that I have to use nullable types and that Kotlin will not prevent me from shooting my own foot if interoperating with Java. My problem is solved.
You are defending Kotlin against attacks I never launched.
What exactly is your point?


#8

Absolutely nothing is broken here.

You declare a map to contain non-null values, then pass that map to another component that does not know about the non-nullability, and which actually does set some values to null. So you are lying to the compiler about what the map can contain:

  • You say the map will never contain null values.
  • The map sometimes does contain null values.

How is the compiler supposed to handle this situation for you? It can’t. There is no way that the Kotlin compiler can force existing Java components to retrospectively honor non-nullable types.

It is up to you to properly handle the null values that get put into the map. So yes, you have to change the declaration to MutableMap<String, Boolean?> and check for null everywhere.

And even if the map only contains non-null values, then you still have to check for nullability (pseudocode):

var map = mutableMapOf("foo" to true)
// Nope, getting a value can result in "null", even
// though all values in the map are not "null"
var value: Boolean = map["bar"]
// Yep
var value: Boolean = map["bar"] ?: false

#9

Meaning I get incorrect warnings by design? That would be even worth than being broken.

I am no compiler guy so I don’t have the ready to implement answers for you, sorry.
Without thinking much it could for example issue a warning when using a non-nullable type to interact with Java.
Or it could disallow the use of non-nullable variables when interoperating with java.
Or it could use Optionals to implement non-nullables.
Or something even smarter than I can come up with.
Remember we are not talking about fundamental natural laws here that the compiler cannot change, we are talking about design decisions.
It’s a design decision to have pseudo-types that only exist at compile time and cannot be handled by the JVM.
It’s a design decision to have these pseudo-types mapped to other types that work different (allow null) transparently.
And it’s a design decision to allow these pseudo types to interact with Java and not analyze these interactions.
At the very least the documentation could have been more clear about the restrictions that come to null-safety when interoperating with Java (which is Kotlins big advertising point: “100% interoperable with Java™”).

And where did I say it should do that?

Hell, I get a NPE when using an elvis operator, does nobody get the irony?

Are you really arguing that it’s ok that this gives an NPE (realcode™)

val v = entry.value ?: false

and this works (realcode™)

val v = if (entry.value != null) entry.value else false

?

And I deeply apologize to the compiler for lying. It was not intended.
I am used to more experienced compilers that don’t get so emotional about a little lie and just tell me to stop the nonsense.


#10

Whether you use a nullable or a non-nullable type to interoperate with Java has been explicitly left as a choice for the developer. Because only the developer can know whether or not the value can be set to null on the Java side.

So the design decision for the compiler has been: The developer needs to choose the correct nullability when interoperating with Java.

As for the “weird” warnings: They are a consequence of you choosing the incorrect nullability.

I feel really sorry for you, because you have to use Kotlin and really don’t like the language. I hope you can convince your team to switch to a better language.


#11

That’s IMHO a bad choice as the compiler actually knows that it cannot be enforced on the Java side. So it’s only any good when interacting with someones own Java code which is a legit use case but the common one is interacting with third party libraries. Coding defensively you must always choose the nullable type. So the default should be to disallow non-nullable types (you could implement a switch to allow non nullable types).
But this is just my opinion and worth to be discussed.

I know. And because I made a mistake it is ok that “weird things” happen?
I hope this is not the philosophy behind Kotlin and only your personal opinion as it would seriously lower my trust in Kotlin.
I think the situation can be improved but maybe I am wrong.

That’s poor form and you know it. Is this the “kind community” I got promised when signing up?
Actually I am an independent developer and can choose my languages freely (except for HTML and JS but nobody escapes these two nowadays).
I choose to port a Java project in development (over 80% done) after having completed a single smaller Kotlin one. I think that says enough about me not liking Koltin.

Not for this project and most likely I will have to switch to an IMHO inferior language.
But after effectively being told to f**k off, I will think about it with a heavy heart. As a professional I cannot afford to rely on a community that does not want me even if the tool is good.

edit: I just realized that you haven’t earned your “Read Guidelines Badge”, yet. So I will take all of this as your own opinion and not the general consensus of the community.


#12

No, because the kotlin java interop specifies, that if java calls kotlin code with null, that NPEs will occur. That’s also why every public kotlin function checks the arguments for null.

But you did not make the mistake in thinking the variable is non-null, because you check it later. You know the variable can be null and you still say it is not. That is why the compiler generates a warning. The problem is, that you on one hand expect the compiler to warn you about the fact that java can return null, but on the other hand you don’t want to change your type because of that. You can’t have both. The moment you declare the type to be non-null there will be a NPE when java calls it with null, as specified by the kotlin specs.

I’m sorry but your argument is: “I can’t be bothered to specify the correct nullability even though I clearly know it (as shown by the null check), therefor I want the compiler to treat my code as nullable when I specify it is not”.


#13

Of cause it is his opinion and not the consensus of the community (although I would guess the community has a similar view)


#14

Yes, I most definitely do know. Do you know that your communication style is very unfriendly?


#15

I try my best but to some extend I know. Can you point out what exactly triggered you?


#16

Well if you look at my code NPE 1 then you see I did actually make this mistake as this is what I did in the Spring project. The other stuff was just playing around when I did the example anyway.
If you look at my second post you see that I was ok with the answers I found (having learned that there is no Kotlin magic that does what I hoped it can). The topic was done for me there.
May I qoute my third post:

Which triggered you and where you conveniently did not quote the last sentence.

As said the null check was not in the original code.
But if you say that that is my argument so be it.
I was under the impression my argument was “the compiler could help to not get NPE 1” but obviously I was wrong there.
I apologize.
I know I put your patience to the test but from all I have learned in this topic it sounds like it is actually ok to use the above posted code and just suppress the warning. As I know this is the only place where the NPE can happen and I test for null there and later it is only touched by Kotlin code I can keep the benefits of using the non-nullable type, right?


#17

Sry, my bad.

Maybe it is possible with a lot of static code analysis, but this would take ages for more complex situations and I am not sure if it is actually possible (otherwise there would be a tool for java doing it). All the compiler checks is that when you declare a variable as nullable you check it first before using it (which still catches 99.9% of all accidental NPEs).
Also the compiler takes your word on nullability when interacting with java functions (which can lead to NPEs). Most of the time this is caught right at the call side as kotlin adds null checks there automatically.

fun foo (bar: NotNull){
    // compiler adds: if(bar == null) throw NPE
}

val f: Foo = someJavaCall() // compiler adds ?: throw NPE

I’m 100% sure on the fact that the compiler adds the first null check and like 80% sure on the other one (I’m to lazy to start up intellij and test it). So the only problem now is generics. Sadly due to the type erasure there is no way of testing whether List<Int> is the same as List<String> at runtime. It sucks, but that’s just the way the JVM works. For this reason it is also not possible to test whether a generic parameter is nullable or not.
I hope this clears up the misunderstanding about what the compiler can and can not do.

Yes and no. There is no guarantee that the compiler will not with one update decide to optimize and remove the if statement with a later version of kotlin, so I’d be careful about that.
The “correct” way of doing it IMO would be to use a nullable type.

for(entry in foo.map) {
    val k = entry.key
    val v = entry.value ?: false
    // all types are non-null now so no inconvenience
}
// or 
foo.map.asSequence().map { (k, v) -> Pair(k, v ?: false) }.forEach { (k, v) -> ... }
// or
foo.map.asSequence().forEach { (k, _v) ->
		val v = _v ?: false
}

#18

If Java reflection can be used to do evil stuff it can be used for good stuff as well.
If I would inspect the possible null value via Java reflection the check would never be optimized away.
Not saying that I am going this route only that that would be an option to keep the non nullable types.
Thanks for your input.


#19

So why you don’t just stick with those compilers and let the Kotlin compiler alone???


#20

Jstuyts, reimagrab and I all 3 were not really following the community guide lines by being not all to nice to each other, as this was a discussion, based on a misunderstanding. I think reopening this after it was settled is not really the best idea, especially as you are not adding anything new to the discussion.

I think it is obvious to most people here that java is inferior than kotlin. And let’s be honest here. The initial question is valid and has a point, as we do run into a NPE (which kotlin supposedly fixes). Understanding when and why it does not work is not as trivial as this discussion shows.