[Idea] Python backend

Multiplatform
13

It’s been a year since project’s inception, so I thought I’ll give you a short update how it goes.

Two contributors, JetBrains’ support

First and foremost, I’m happy to share that Sergei “SerVB” Bulgakov joined the project! He works for JetBrains, on Projector project, and he decided to devote 20% of his time for Kotlin/Python. Let me put it straight: if not Sergei, the project would be either dead or at most 20% of what it’s now. Every Friday he pushes stuff forward, step by step. Thanks to the fact that Sergei works for JetBrains, it’s also easier for him to get in touch with the Kotlin team. In fact, the Kotlin team has been helping us in a form of advising what patch to take during implementation, hinting how stuff works in other backends, and so on. You can be a witness of this cooperation on #python-contributors Slack channel.

Progress

Does it work good enough to be usable? Well, we’re not there yet. However the progress we’re making is systematic and looks promising. As our main metric how far we are, we assume the number of passing platform-agnostic box tests. Box tests (short for black-box tests) are simply end-to-end tests of the compiler, taking a sample Kotlin code that does something and is expected to return "OK". They are a great help and allow us to develop the new backend in a somehow test-driven way. You can see example box tests in Kotlin’s main repository.

Here’s a visualization made from our project’s git history, as of November 2021:

Box tests history

To see the current plot, click here.

Talk is cheap, show me the code

Basic language features are implemented, but the standard library is almost unusable yet.

Here’s a test that shows that integers can be compiled and work the same as in Kotlin. It also shows basic stuff like calling functions, recursion, when, simple boolean logic, and other.

Kotlin code:

fun isPowerOfTwo(n: Short): Boolean = (n.toInt() and (n - 1)) == 0

fun factorial(n: Int): Long = when (n <= 1) {
    true -> 1
    false -> n * factorial(n - 1)
}

fun numberOfCombinations(n: Int, k: Int): Long = factorial(n) / (factorial(k) * factorial(n - k))

fun sumOverflowDemo(a: Int, b: Int): Int = a + b

compiled to Python (omitting standard library):

# ... 13300 lines of Kotlin stdlib Python code...

def isPowerOfTwo(n):
    return n & (n - 1).__add__(0x8000_0000).__and__(0xffff_ffff).__sub__(0x8000_0000) == 0

def factorial(n):
    tmp0_subject = n <= 1
    if tmp0_subject == True:
        tmp = 1
    elif tmp0_subject == False:
        tmp = n * factorial((n - 1).__add__(0x8000_0000).__and__(0xffff_ffff).__sub__(0x8000_0000))
    else:
        noWhenBranchMatchedException()
    
    return tmp

def numberOfCombinations(n, k):
    return (factorial(n) // (factorial(k) * factorial((n - k).__add__(0x8000_0000).__and__(0xffff_ffff).__sub__(0x8000_0000))).__add__(0x8000_0000_0000_0000).__and__(0xffff_ffff_ffff_ffff).__sub__(0x8000_0000_0000_0000)).__add__(0x8000_0000_0000_0000).__and__(0xffff_ffff_ffff_ffff).__sub__(0x8000_0000_0000_0000)

def sumOverflowDemo(a, b):
    return (a + b).__add__(0x8000_0000).__and__(0xffff_ffff).__sub__(0x8000_0000)

Python consumer:

from compiled import isPowerOfTwo, factorial, numberOfCombinations, sumOverflowDemo

print(isPowerOfTwo(32))
print(isPowerOfTwo(33))

print(factorial(5))

print(numberOfCombinations(4, 3))

print(sumOverflowDemo(2 ** 31 - 1, 2 ** 31 - 10))

Output:

True
False
120
4
-11

To give you a clue what kind of simple things don’t work yet and why, compiling and executing such Kotlin code:

fun test(): List<Int> {                                                 
    return listOf(1, 2, 3)                                              
}

results in a mysterious

NameError: name 'kotlin_Any_' is not defined

It’s because listOf is a part of the standard library where class hierarchies come into play, and apparently something about the base class Any is not yet implemented. It’s a short, basic function, but actually using complex machinery under the hood.

See more end-to-end tests and box tests report to learn more.

If you’re curious what the standard library compiled to Python looks like so far, see here.

Development workflow

Each commit is checked with CI and tested against the box tests. You may ask: how come the test come up green if the backend is not yet ready? Well, we allow the tests to fail and still have green overall CI result, but in return we maintain a file with a list of failed tests together with a file with more details about test results. It does add some complexity to the PRs, but thanks to this we can manage regressions. A typical contribution that covers new features isn’t checked “if the tests still pass” or “do the new tests pass” because we don’t add new box tests, but instead: “are there some box tests that failed before this change and don’t fail after”.

To make this whole ceremony of managing extra files painless, we have such Kotlin script that generates everything for us, including the above plot.

See python/README.md#development for more details.

Can I try it out?

Sure! It’s enough to:

  • clone the repo
  • run ./gradlew dist
  • compile your Kotlin file to Python: dist/kotlinc/bin/kotlinc-py -libraries dist/kotlinc/lib/kotlin-stdlib-js.jar -Xir-produce-js -output compiler_output.py your_kotlin_file.kt
  • examine compiler_output.py and maybe consume it from some Python module

Please be prepared to be disappointed by the lack of basic features. It’s still too early to even experiment with it in pet projects, but it’s fine to check out the project’s progress on your own.

What’s next

Well, we are going to push it further bit by bit, hoping that we’ll keep the pace. Any help is greatly appreciated - the dev workflow described above makes it pretty simple to contribute. The goal for now is providing unidirectional Kotlin to Python compilation. You will be able to write simple console apps in platform-agnostic Kotlin running with the Python runtime, and simple libraries consumable from Python.

Concrete next steps:

  • create Python-specific implementation of Kotlin’s standard library. We now piggy-back on JavaScript’s one and it starts to be painful
  • find a simple, real-life platform-agnostic Kotlin library and work towards compiling it for Python
  • if we start dreaming: work with the authors of lets-plot. Currently, another solution to achieve “write in Kotlin, run with Python” is in place

Any sign of your support and/or interest in this project is a great motivator for us :slight_smile:

In case of any questions, feel free to reach us at #python or #python-contributors, depending if you’d like to use or contribute to the project.

13 Likes