[Idea] Python backend

What do you think about having Kotlin + Python interop, just like we have for JS or Native?

Python is extremely popular, and getting even more popular. It’s now industry standard in machine learning and data science. It’s not about superseding Python with Kotlin, but rather giving Kotlin an alternative for those who want to reuse their Kotlin knowledge and reuse existing Kotlin platform-agnostic logic.

About a year ago I created an issue proposing this idea: https://youtrack.jetbrains.com/issue/KT-34074 - click to see more details about the idea and example use cases. Someone from JetBrains has even commented there, but he doesn’t seem to be speaking on Kotlin team’s behalf, but rather as a potential client.

This idea has been sitting in my head for quite a while, so I’m curious what’s your take on this - if you also see benefits of it, and if someone has already considered it. I’m also wondering if someone from the Kotlin team has performed a business analysis of such interop (explicitly deciding not to go into this direction for now), or it’s implicitly not being looked at.

Best,
Piotr

2 Likes

More backends are a great idea. It’ll definitely have to wait until the compiler overhaul is done.
I believe there’s already an official WebAssembly backend in the works–and of course, the remaining old backends are being rewritten to the new frontend.

As far as other backends, a .NET target has been a popular request. There was some talk about working on (and funding?) a .NET backend by the community. I’m hoping it will be possible for the community to adopt new backends for Kotlin independent of the officially supported targets.

Python would be nice–if the community is able to make some headway on it first I suspect it would really strengthen the odds.

2 Likes

Do you have any insight into when the overhaul is done, and why actually is it needed? I thought about putting together some Python interop PoC as soon as IR can be emitted for common Kotlin (IR -> Python, to be used from Python-based entry points), and AFAIU it’s already possible given that some backends already use it experimentally.

I tried analyzing the code base, looking at how it’s done for JS, but I feel a bit overwhelmed :slight_smile: If you have some docs on the internal architecture of the compiler or anything that would be helpful… I wish creating a new backend was as simple as implementing a well-defined compiler API, or at least following well-defined steps described somewhere. Well-documented IR would be the meat of that API, I presume.

Regarding .NET, I thought that it got disbanded pretty quickly, looking at https://youtrack.jetbrains.com/issue/KT-1287. But looking at this forum, there were several topics on this forum indeed.

1 Like

Yeah, it might I bet it would be possible to use the current Kotlin IR and experiment with Python. I have no clue about documentation though–it’s out of my purview.

By all means, I’d love to see community efforts. The reason I said I think it’ll have to wait is that I was assuming the Kotlin team would already be tied up with the other backends. Also I thought the IR was still experimental. If it’s community-driven and people are okay with building on an experimental state then all bets are off.

1 Like

I took a glance how IR -> JS translation is done. It’s actually not that hard to grasp. Here are my notes for someone that wants to tackle this (might be me one day :slight_smile: ):

  • the most interesting Gradle module: compiler/ir/backend.js (to see its logic, jump straight to compiler/ir/backend.js/src/org/jetbrains/kotlin/ir/backend/js)
  • the top-level file seems to be compiler.kt, with two functions: compile returning CompilerResult that contains val jsCode: String?, and generateJsCode. IR is the input, JS code is the output
  • going deeper, see transformers/irToJs/IrModuleToJsTransformer.kt and other ...ToJsTransformer.kt files. They seem to be the places which perform the actual mapping from IR entities to JavaScript abstract syntax tree (org.jetbrains.kotlin.js.backend.ast.* entities)
  • the mapping of JS AST to the actual JS code is common for both the old and the new IR backend (see js/js.ast Gradle module). It happens in JsProgram's toString(), inherited from AbstractNode. The actual implementation sits in JsToStringGenerationVisitor

Next steps: compile the compiler/ir/backend.js module (after a small modification, to be sure that the modified version is used instead of the prod Kotlin compiler) and use it with some example Kotlin code. Once it works, we can gradually experiment with making adjustments towards Python. Maybe even the “IR -> JS AST” layer can stay as is for now (I see some similarities between JS and Python and we could take advantage of them), and only “JS AST -> JS code” layer can be adjusted to generate Python.

1 Like

Cool stuff! If you get some momentum or a hello-world for Python-- be sure to share on GitHub. I bet a lot of people would be interesting in playing around with any progress or learning for other compiler customizations.

Sure, experimenting here, see latest commits: https://github.com/krzema12/kotlin-1/commits/python-backend

For

fun pythonTest() {
    println("Hello world")
}

and a command dist/kotlinc/bin/kotlinc-js -output kotlin-python/out.js kotlin-python/python.kt (uses the classic backend, not IR) so far I got something that still isn’t valid Python (not much missing, though):

def (_, Kotlin) :
   println = Kotlin.kotlin.io.println_s8jyv4$
  def pythonTest() :
    println('Hello world')
  
  _.pythonTest = pythonTest
  Kotlin.defineModule('out', _)
  return _

but it’s encouraging to see some progress at least in terms of working with the Kotlin compiler code base :slight_smile: And that creating a custom backend is not rocket science.

This little experiment leads me to a conclusion that it would be better to implement Python’s AST from ground up, to avoid fighting with some of JS peculiarities. Adjusting JS parts to work like Python is IMO not the way to go long-term. I’ll push this experiment just a bit more forward and then will think how to implement the Python backend the proper way.

3 Likes

I decided to start with implementing data structures to express Python AST. From this point, both integration with Kotlin’s IR and generating Python code can go in their own paces. I won’t spam here too much - if anyone’s interested in the progress, see https://github.com/krzema12/kotlin-1/commits/python-backend, especially the dedicated README with progress. I’ll describe my approach below in case someone comes up with a better idea - any feedback or help appreciated.

The starting point is Python’s grammar from CPython project described using Zephyr ASDL. I want to write a tool that parses ASDL and generates Kotlin’s data classes, sealed classes and whatever else needed. I googled for an existence of such ASDL -> Kotlin generator - didn’t find anything. Having such tool, the process of generating code to express the AST becomes data-driven instead of maintaining it by hand. I won’t focus on any particular Python version, I want it to be Python version-agnostic. I’ll test the generator on CPython’s master branch (now 3.9) and some earlier 3.x versions. This milestone is complete once we can take any Python code and express it using Kotlin entities.

4 Likes

That’s very interesting, is there a way to get notified by your progress? I’m following releases on your repo, but that’s probably not the solution. Maybe you could create a channel on the Slack?

I’m not planning to create GitHub releases because I create multiple little commits. I thought that if you watch a given repo, you can mark to get notified about all changes, but not sure if commit made on a non-main branch emit notifications.

I promise to post updates in this thread once I reach major milestones. I’ll take a look later at creating a new Slack channel in Kotlin’s space, it may be indeed a nice place to have side conversation.

I created a #python channel in kotlinlang.slack.com.

1 Like