Custom String Templates

Language Design
1

Scala has a great feature that allows for adding custom string interpolation handlers, signaled by adding a prefix in front of the opening quote. They have built-in versions s"" and f"" (the latter which allows adding %.2f style formatting strings after the ${}).

But they also allow custom prefixes, which my company has used to very good effect for building a powerful i18n framework and SQL layer. There are a number of ways to expose it. Here is one suggestion:

Allow adding a function in a specific spot, e.g.:

fun templateHandler(prefix: String, parts: Array<String>, args: Array<Any>): String

Where prefix is the thing that comes before the opening quote, parts are the string literals between interpolated expressions, and args are the interpolated expressions.

If the template starts or ends with an interpolated expression, parts should respectively start or end with an empty String so that the every-other pattern of the parts and args can be relied upon.

This would allow building things like:

  • json"a: $a" (where a can be an array and will get formatted correctly)
  • sql"select 
" (and do proper escaping of arguments)
  • tr"You have a balance of $count credits remaining. " (and get translated)

Because the existing string interpolation functionality does not have a prefix concept, it should be possible to add this in a backwards-compatible way and without impacting performance for the existing approach.

It is essential that the handler function could be non-static so that it can access and mutate relevant state in an object instance (i.e., access the this pointer), and so that different handlers can be implemented in different classes.

There is a tension between requiring a class to implement an interface to enable this vs. allowing static implementations. Ideally it would be possible to do both, but allowing access to instance state is very important for the use-cases that we have encountered at my company.

And it should not be required that the handler return a String, as many use cases have side-effects but do not return anything.

(Back-story: we are frustrated with Scala compile times and are considering a switch to Kotlin; this is a must-have feature to enable the switch.)

5 Likes
2

Is it possible to combine templateHandlers?

3

I don’t understand the question. I think it should be best understood as syntactic sugar that maps:

abc"Interpolate this: ${a}"

to:

templateHandler(“abc”, arrayOf<String>("Interpolate this: ", “”), arrayOf<Any>(a))

And wherever the call is made, it will take whatever templateHandler method is in scope, whether it be a class function or some top-level function that is accessible. Basically, the semantics would be whatever would get called if the string template where literally replaced with the translated call in the code file.

Alternatively, in order to allow for different return types for each prefix, you could have a naming convention such as:

fun templateHandler_abc(parts: Array<String>, args: Array<Any>): Unit
fun templateHandler_xyz(parts: Array<String>, args: Array<Any>): String

Then the compiler can make sure that there is an available handler function to match the prefix present in the expression. That is more in line with how Scala does it (though there are still significant differences).

4

I suggest you to find a Kotlin way or remain on Scala.
It is really difficult to get this feature soon.

Please consider to find a solution to your problem, I don’t think that custom string template is a game changer.

Regarding your proposal, can you explain better how it works?
Example:

sql"SELECT count(*) FROM $table WHERE $attribute = $value"

json"name: $userName, friends: $friendNameList"

5

Our main make-or-break use case is our HTML builder with internationalization (i18n). Similar in some ways to the sketch provided at:

https://kotlinlang.org/docs/reference/type-safe-builders.html

But instead of using the + operator to append the string literals, we use custom Scala string interpolators.

This allows us to generate valid ICU4J MessageFormats, complete with type-safe translation-and-type-aware interpolation of the ${} content. Interpolated content can be fully machine-rendered (like numbers, currency, or dates), or it can be a mixture (such as plural forms, where the machine picks the plurality based on an input number, but the translators provide the inflected forms).

In this way, we are able to code HTML in-framework, using the same type-safe language that we use for everything else, all while getting i18n essentially for free (free from a coding perspective; we pay translators to translate the ICU MessageFormat phrases).

At the end of the day, we make an edit in Scala and then deploy. The ICU MessageFormats get collected in a production database and our translators are notified. They complete the translations in 24 hours and our site is back to full i18n support without any additional effort on our part compared to an English-only site. This has allowed our 2-person team to run half a dozen sites fully translated into 8 languages other than English.

This is in contrast to the crazy systems of externalizing all phrases that need to be translated and writing them in native ICU MessageFormat, which is extremely difficult to type correctly without errors. Those systems also make the page content essentially unreadable in the code because all the text content has been externalized.

Why is it difficult to get this feature soon?

6

Generally a feature like this has to first be introduced as a KEEP. Then the kotlin team has to decide to adopt it and it needs to be implemented. Even if the kotlin team instantly decides to implement it, it will probably not happen before the compiler backend rework is done, so at best it will be available in an experimental state about halfway between 1.4 and 1.5, probably not before 1.5.

1 Like
7

This is the first time I hear about the feature that you are requesting. Hence I believe there is not so much popular demand for it.

There are lots of other features for which the popular demand is very high, and which are already waiting to be implemented for years. Literally years.

Even if the feature request is accepted, I would keep expectation low that it will ship sooner than 2 years from.

8

This feature is very popular among scala developers, some of which, like me, are considering moving to kotlin. Also, some C++ geeks, like me, know that such thing could really make a day in some cases. Only few of us consider using kotlin instead of scala, though as for me kotlin may have brilliant future if it will support decent set of language features like this one.

The compile time support for custom string interpolation is vastly underestimated. While almost any scala developer uses it, most of them use the ready libraries and even don’t know what language feature they use. All these things, like regular templates s"Hello $name", regexps r"Hello,?\s+${fisrtName}", andfantastically convenient anorm’s SQL builders SQL"select * from some_table where first_name=$name" are based on the complile-time custom interpolation. Lets see this example more in depth.

With kotlin, in no way it is possible to make such a simple, effective and expressive SQL builder based on natural looking SQL string:

 SQL"select * from persons where first_name=$first and last_name=$last"

Why? Because (right) SQL support requires not the string interpolation, but _detecting and separating the list of parameters without loosing type information, and building (and caching prepared) statement with it. Text interpolation just does not work, is ineffective and dangerous: somebody could just forget to escape interpolated data creating infamous sql injection vulnerability. E.g. client library needs to separate ${} values from the template to process them as is. With current kotlin we are bound to ugly in that case builder pattern, something like

SQL().s("select all from persons where first_name=").p(first).s(" and last_name=").p(last)

This is ugly, though effective. Because here the programmer “compiles” the interpolation to kotlin, while scala happily does it for us.

This is, actually, main reason why our new server-side projects are still started in scala. Because we love, need and use directs SQL statements widely (after tenth years of pain with ORM-like wrappers that just eat our and server time). And with kotlin our database code will turn to a mess.

I should admit that in many many cases builder/extensions pattern is enough effective. I appreciate things like “^Foo(:?bar)?”.re, but certainly we (most of) really miss of re"^Foo(?:$bar)?" that will implement correct escaping. The existing alternative, “Foo(?:${Re.escape(bar)})?” is times less readable and error-prone. For example, forget to escape in interpolation, and you’ll have a certain hidden bug that could easily escape unit testing.

This is a classic case when the feature could be relatively easily added to the language, as it will not break compatibility and is not overly hard to architect and implement. I’d certainly use kotlin in the next backend then.

1 Like
9

Thanks for the explanation. I’m not experienced with this feature of Scala. Just trying to get on the same page to know what’s up.

One part that I suspect could be separated from the request is the compile-time aspect. It seems to me that the main lacking part of normal string interpolation is that it is interpolated at the call site. You can’t call something like:

// Want to do this--allow someFunction to reason on templating instead of passing plain old string:
someFunction("interpolated string but not at the call site; $var1 $var2!")

// But instead you must do something like the following
someFunction("plain interpolated string %s %s!".doTemplating(var1, var2))
// or
someFunction("plain interpolated string ${var1.doTemplating()} ${var1.doTemplating()}!")

I imagine this could be done in a few ways:.

  • new string type in Kotlin (maybe ‘’‘my string’‘’) that allows inspecting and modifying how the interpolation is done before being toString’ed
  • new keyword modification to vararg (something like fun someFunction(template vararg text: String),
  • new operator function (to allow something like operator fun SqlTemplate.template(source: String, vararg params: String).

^ I’m not advocating for any of these, just trying to say I imagine the custom templating being done at runtime.

1 Like
10

Yes, the compile-time aspect is secondary. It would be nicer for performance reasons, but all of the use-cases that our team has would be just as well served by a runtime solution.

The true key is the syntactic clarity and type information at the point of applying the template.

If you can map:

abc"Interpolate this: ${a} and ${b}."

to:

templateHandler(“abc”, arrayOf<String>("Interpolate this: ", “ and ”, "."), arrayOf<Any>(a, b))

Then you are able to do what is needed.

I believe that this is a widely used feature in Scala and there are numerous DSLs built around it. We have built four internally-used DSLs based on this capability and it leads to much more concise and safe code than any other approach to this problem.

2 Likes
11

If custom string templates would be available, how would you avoid the following problem:

Consider the following use case:

val searchTerm = "something"
"select * from Book where name = ${sqlEscape(searchTerm)}"

This could be written as:

sql"select * from Book where name = $searchTerm"

I usually keep constants with table names and column names so I won’t directly type the “Book” table name:

object TableNames
{
    val Book = "Book"
}

"select * from ${TableNames.Book} where name = ${sqlEscape(searchTerm)}"

The result will be

select * from Book where name = "something"

However, if I do

sql"select * from ${TableNames.Book} where name = $searchTerm"

The result will be

select * from "Book" where name = "something"

Obviously I don’t want the table name to be escaped into a string.

I like this proposed feature, but it can introduce a bit of ambiguity, it removes explicitness.

12

Well that depends on how your sql function is structured. You could, for example, add a backslash before a variable so that the function knows that you don’t want to escape that table name into a string. I’m sure that the Scala community has already figured out how to deal with all of this. Maybe we could even have a slightly different syntax when you want to pass a variable instead of a string (i.e. maybe @{Terms.searchTerm} for example which tells the compiler that you want that to be passed to the vararg parameters instead of being directly interpolated into the string. Then your example will look something like this:
sql"select * from ${TableNames.Book} where name = @searchTerm"
and it would work as expected)

Note: I’m not saying that the @ sign should be used for that; I just used it as an example to demonstrate the ability to have a different syntax for this specific feature.

13

This all just sounds really dangerous and prone to introducing SQL injection attacks. Why not just use a type safe DSL builder (which knows which parts needs to be escaped in which way) or plain prepared statements instead of String manipulation.

I find this proposed feature mainly useful for translating Strings. Kotlin String templates are really unhelpful if you want to to put out nicely formatted and localizable text and something like this would probably help a lot.

1 Like
14

It is really powerful for language translation in a web framework. You can do things like:

trs"There were ${count} cute ${TrPlural(count, "dog", "dogs")} at the park. "

or:

trs"The moon phase is ${TrChoice(phaseIndex, "new", "waxing crescent", "first quarter", "waxing gibbous", ...)} on ${TrDate(date, DateFormat.YEAR_MONTH_DAY)}. "

And have that generate the required ICU translation phrase, add it to your database as required, fetch the required translated phrase with interpolators, and execute the template to get the desired localized output.

The examples above really only scratch the surface. We are a two man team running multiple websites, all translated into 8+ languages and we barely give any thought the translations because our framework is so powerful and seamless.

Scala makes it possible because they support custom string interpolators. Kotlin should add the same functionality because it is extremely powerful and leads to really clean nice syntax.

We use it for several other purposes. The applications abound.

1 Like
15

Why do those examples need custom templates? (I’m probably missing something obvious, but couldn’t you do exactly the same in Kotlin just by defining functions TrPlural() &c?)

16

When a trs"
" is invoked, it:

  1. Translates the entire Scala interpolation template into a valid ICU translation phrase (which has its own interpolation syntax), and either finds it in the DB or adds it to the DB.
  2. It finds the translated target-language version of the source-language ICU phrase once the translator has provided it.
  3. It fills the template using the provided type-safe parameters, and including complex args like TrPlural objects, and already translated interpolants.

So almost none of that can be done by just invoking on a method on the arguments. The only real alternative is to create the content as valid ICU phrase units, and fill them using the conventional function calls that look like:

ICU.translate(locale, "This is an ICU template with interpolants {0}, {1}, etc...", arg0, arg1, ...)

That works fine, but it is a less fluid approach (it is template-then-args, rather than an inline interpolation style), it is typo-prone, doesn’t take full advantage of the compiler for error-avoidance, and it gets really messy when you start using it at scale.

I’m going to bow out of this discussion now. We’d love to use Kotlin over Scala because it is much faster to compile (which is the main knock against Scala), but this feature is a deal breaker and it is clear from this thread that it won’t be coming anytime soon. Good luck to the other interested parties!

17

It seems that custom interpolation in scala needs “implicit” keyword, which is not wanted in Kotlin.
One advantage of ICU.translate syntax is that it can work with variable :

fun foo(template : String)  {
   val tr = ICU.translate(locale, template, ...);
...
}

which is not case with scala string interpolation (nor with kotlin String with args)

18

Sql / Html / Other 
 Builders with custom string interpolation are very powerful.
Type safe. Type safety is safe against injection vulnerabilities.

for example, you can use

val searchTerm = "word"
var arge = 12
val tablName = Esacped("Book")

sql"select * from $tableName where name=$searchTerm and age = $age"
 >>> QueryAndValue("select * from Book where name=? and age = ?", listOf(searchTerm, age))

This is easier and safer than the following example.

Sql("select * from ${tableName} where name = ? and age = ?" , searchTerm, age).execute()

Sql("select * from ${tableName} where name = ).p(searchTerm).sql(" and age=").p(age).execute()

implementation is maybe

fun sql(parts: Array<String>, args: Array<Any>): QueryAndValues{
  var i = 0
  var query = ""
  var values = mutableList<Any>()
  loop{
    if(i >= parts.length) break
    query += parts[i]
    if(i >= args.length) break
    when(var a=args[i]){
      is Escaped -> {
        query += a.value
      }
      else -> {
        query += "?"
        values.add(a)
      }
    }
    i++
  }
  return QueryAndValues(query, values)
}

and if, string interpolation handler signature is fun sql(parts: Array<String>, args: Array<Int>): QueryAndValues,
If you use it sql"aaa + ${"string"}", you will get a compilation error.

19

Few concerns.

  • custom interpolation could easily be used to ensure no SQL injection could be done. As it is done Scala’s ANORM

  • it neither relies not requires “implicit” keyword nor require it to use its functionality

  • the compile time parsing is a key. No runtime parsing could give neither comparable speed nor readability nor functionality. I mean:

Scala way url"https://acme.com/items?matching_name=${namePart}&limit=${limit}" is clean and nice.

Current Kotlin way, something like urlescape("https://acme.com/items=?matching_name={}&limit={}", namePart, limit) is ugly and hard to read. We can’t use even ? here :wink:

It bounces us back to C times, when format precedes data list. It is what all languages, Kotlin include, tends to replace with string templates. Now make one more little step and let us write our own modern interpolators. Not in 1980’s printf style :slight_smile:

Take a look of SQLDelight. They have to parse SQL grammar in a plugin to get the same results they could easily achieved having custom interpolations. And, alas, it is pretty unusable as parsing complex SQL grammar needs a enormous parser


The most intriguing as for me is a strength of reaction against such a useful and simple to code feature. Listen, colleagues, if some feature was implemented in scala, it does not automatically means it’s bad :wink:

20

What is currently possible, for example, and not so very “ugly”:

    val id = Column("id")
    val user = Column("user")
    val sql = buildSql { "select ${+id},${+user} from table" }
How: 
class Column(val name: String)
object SqlBuilder {
    operator fun Column.unaryPlus(): String = "\"$name\"" //quote name
}
inline fun buildSql(block: SqlBuilder.() -> String): String = SqlBuilder.block()