Hello
I have used IDEA to convert the following Java Spark code to Kotlin (latest version of IDEA and Kotlin 1.3.31)
Here is Java:
public static void getGuid() {
SparkConf conf = new SparkConf().setAppName("Java UDF Example");
SparkSession spark =
SparkSession.builder().enableHiveSupport().config(conf).getOrCreate();
spark.udf().register("getguid",
() -> java.util.UUID.randomUUID().toString(),
DataTypes.StringType);
}
public static void ctof() {
SparkConf conf = new SparkConf().setAppName("Java UDF Example");
SparkSession spark =
SparkSession.builder().enableHiveSupport().config(conf).getOrCreate();
spark.udf().register("ctof", new UDF1<Double, Double>() {
@Override
public Double call(Double degreesCelcius) {
return ((degreesCelcius * 9.0 / 5.0) + 32.0);
}
}, DataTypes.DoubleType);
}
Here is converted Kotlin:
fun getGuid() {
val conf = SparkConf().setAppName("Java UDF Example")
val spark = SparkSession.builder().enableHiveSupport().config(conf).orCreate
// this does not compile, Overload Resolution Ambiguity
spark.udf().register("getguid",
{ java.util.UUID.randomUUID().toString() },
DataTypes.StringType)
}
fun ctof() {
val conf = SparkConf().setAppName("Java UDF Example")
val spark = SparkSession.builder().enableHiveSupport().config(conf).orCreate
spark.udf().register("ctof",
{ degreesCelcius -> (degreesCelcius!! as Double) * 9.0 / 5.0 + 32.0 },
DataTypes.DoubleType)
}
Here is what I think I need:
public void register(java.lang.String name, org.apache.spark.sql.api.java.UDF0<?> f, org.apache.spark.sql.types.DataType returnType) { /* compiled code */ }
or this ?
public <RT> org.apache.spark.sql.expressions.UserDefinedFunction
register(java.lang.String name, scala.Function0<RT> func, scala.reflect.api.TypeTags.TypeTag<RT> evidence$1) { /* compiled code */}
In other words nothing goes in and function returns 1 value as out.
I used method reference instead which worked:
spark.udf().register("getguid",
UUID.randomUUID()::toString,
DataTypes.StringType)
Can anyone assist me how to migrate this “ambiguous” call please, I am new to Kotlin.
Can I use another way to do achieve the same result without using method reference?