Imagine if you could extend Kotlin grammar directly in the code. In c-like languages such extensions may be more or less precisely identified with macro definitions. But macros are ‘stupid’ in the sense that they are straightforward, dont’ consider any semantical or syntactical fitness (you can easily break type safety or syntactical correctness with such macro-substitutions).
What I’m speaking about may be exactly expressed as: we extend the grammar of the language inside code. Of course, to do it we must expand the core grammar with special construction for the description of the new grammar rule and it’s meaning. And the semantics of such construction is straightforward: given an AST node of newly defined kind, we transform the tree back to the core syntax, using the definition of the new syntax construction.
The main point of doing this: such things as syntactic sugar would be no more tied strongly to the core language, but may be left for the consideration of developers.
So, pros and contras of this feature:
- syntax is becoming very flexible
- significant part of discussions about the language design will move to the standard library design
- some syntactic sugar features of the language may be transferred from the core to the library, so core language will become smaller
- in case of misuse this feature makes it possible to break the language syntax.
- even in case of not so responsible usage it may introduce undesired ambiguity and inefficient parsing
- performance overhead for the dynamic nature of the parser
The technical side of this proposal. If I didn’t implement something alike, I wouldn’t have raised such discussion. I made a parser for the language Russell, which has no built-in syntax of expressions. The parser for the syntax of expressions is generated on the fly, while parsing the main source. The size of the source files for Russell is quite large: about 150 Mb, the syntax of expressions is also quite complex: about a thousand of rules, and the parsing of such amount of source takes it all in all about 2 minutes (4 core CPU).
Some info about my implementation of such a dynamic parser. At first I tried naive LL parser, but it failed with grammar ambiguity. Then I made a huge effort and implemented an LR parser, which was very fast to run, but very slow to generate - so unfortunately, it was unacceptable - we can’t wait for hours to get the source compiled. At last, I ended with a version of LL parser which could parse all of the expressions in reasonable time, despite the fact of ambiguity.
Main problems with implementation of such feature:
- All parsing subsystem of Kotlin must be completely revised
- Operator precedence rules of the grammar: yet I don’t know how to implement it in the parsing algorithm which I used for Russell.
- critical - the speed of parsing. It’s clear, that the dynamic parser will be slower then the highly optimized hand-written parser for a fixed grammar. The main point is the degree of this gap in performance.
As for me, I’m reservedly optimistic with these problems, but it’s clear, that it’s a huge work to implement such feature.