Libraries in “Kotlin -> Javascript” projects


#1

Libraries in “Kotlin -> Javascript” projects.

The purpose of this message is to share some thoughts about current state of supporting libraries for developing Javascript applications using Kotlin.

For now there is only one such library --- kotlin-jslib.jar, which contains subset of standard kotlin library, translated to Javascript.

The first step in order to extend support for different libraries of these kind is to implement the following requirements.

Library is a jar-archive, which contains kotlin sources (in the future it will be replaced with serialized descriptors in binary format) and javascript files (result of translation).

META-INF/MANIFEST.MF contains some specific attributes in order to determine the kind of library and javascript module name.

CLI & buildtools

Console compiler has  -libraryFiles option, which value is a comma-separated list of paths to jar-files or folders. It is assumed that each folder is really an expanded version of some valid jar-archive library, in particular, it should contain manifest file META-INF/MANIFEST.MF.

Libraries could also be specified as dependencies for maven project and added to IntelliJ IDEA project.

IMPORTANT NOTE: console compiler, ant kotlin2js task and kotlin maven plugin doesn’t copy javascript files from libraries, their work is only to translate given kotlin files to Javascript (though this decision should be investigated).

Console compiler and build tools will support an option for generating meta information.

IntelliJ IDEA

It is possible to configure the following additional option for Kotlin Compiler in IntelliJ IDEA ( Settings/Build, Execution, Deployment/Compiler/Kotlin Compiler/Kotlin to Javascript section):

output directory for runtime library files (default value is “lib”, relative to the directory which contains output file).

When IntelliJ IDEA compiles project, each library processed in turn, and all javascript files are copied to output directory for runtime files.

For each library we have semantically two step process:

1) recursively copy js files from library directory or archive except META-INF directory with saving directory structure;

2), recursively copy js files from META-INF/resources subdirectory of library directory or archive with saving directory structure. This could be useful for JVM-based containers (see http://www.webjars.org/documentation#servlet3).

The next step is to implement binary serialization for descriptors. Also there is a possibility to include descriptor metadata into corresponding javascript file, so the whole library could be distributed as one javascript file.

Questions and comments are welcomed.


#2

Sounds reasonable.  What is the plan for sharing code between Kotlin JVM and JS?  So I have some logic that acts upon model classes and I want to reuse them.  Currently the code in Intellij is compiled for one or the other, but not both.  Will it be possible to have shared code?  From the command-line I'm guessing I just compile twice with different dependencies, but that leaves IDE support lacking (and I like using my IDE to be productive), and also Maven/Gradle not being able to work.  I see projects like Abesto's Kotlin-Pixi doing manual Gradle work to make up for lacking toolchain support, and it would be good to have a workflow smoothed out early.


#3

Also, what build of IntelliJ + Kotlin has the support you mention, 14.0.1 + 0.9.999 does not have the javascript jar library support as far as I can see.  


#4

About the binary serialization format:

  • why not a foo.d.kt similar to typescript definition files?
  • why not a typescript definition file?  (they have many available view DefinitelyTyped, no sense re-creating that world)
  • why not a dart external definition file? (they have the same problem, sucks to talk to JavaScript or other libraries with that barrier)

It would be really nice if dart, typescript, kotlin could interop with their definition files … agree on something, not sure if binary or human readable.  

A binary format that could be used by all is nice, but it is unlikely the other languages would adopt it.  Also a binary format doesn’t allow you to easily view what is available from the library.  So from “learnability” if I receive a library without original source, my IDE will need to be able to “de-compile” into something that looks like Kotlin to me so I can view it (similar to java class file decompilation).  Browsing via code-completion isn’t always as productive as viewing the whole class quickly as text.

Things to think about.


#5

About the JavaScript + JVM model more.  I think this needs thought of before designing this library format completely.

Here are some use cases:

I want to create a library that talks on web sockets, has my own custom protocol, uses Snappy compression (my own implementation), and has an top level communication API.  I want to use this both in the browser and on the server.  I do not want to implement the protocol twice, nor the compression, nor the communication logic.  I also don’t want to do crazy build tricks to copy and mangle files on build to make things work, nor have copies of the same source in two trees.  Instead:

Communication API (Kotlin compiled to both platforms) =>
         uses Snappy (from third party Kotlin code compiled for both platforms)
         uses Protocol (Kotlin code compiled for both platforms)
                  uses Web Socket abstraction based on traits
                      uses Web Sockets from HTML5 Browser
                      uses Web Sockets from Some Java Library

So I have 3 pieces of shared code, with one needing to fork to libary A or B depending on platform (web sockets).   How do we make this work?

  1. Libraries like the Snappy one need to be able to be compiled from one code base to two targets, JavaVM and JavaScript and delivered in a way that supports both at same time with common API
  2. Libraries like the Communition API are similar, compile to both platforms using JAR's compatible with both for things I use
  3. Protocol is more complicationed in that it needs to fork at the platform level but re-use as much code as possible, it does things like heart beats, re-connections, and more
  4. Web Socket abstraction needs to wrap two different platform API's while presenting one unified API upwards

This is actually a real use case for us right now (we do Dart client-side, Dart server-side, Kotlin server-side and in IDE plugins all talking) but without the ability to use common libraries, so we reproduce code in different languages.  Or if in Kotlin would need to do tricks with swapping in files or symlinks then recompiling.  

So if Kotlin is going into this game, let’s not be at the same level as Dart or Typescript with leaving this problem for every programmer to solve.  Let’s make it easy to work across client and server with shared code and shared libraries and do so happily.  It’s hard, but that’s why we are here, to finally solve harder problems with a better language platform like Kotlin.

Ok, another use case, and this one mixes things in one project:

I have a Kara web application, and in my DSL for generating the HTML I have an EMBEDDED script tag with DSL that is Kotlin-for-JavaScript, so do we mix platforms within one source file?  Because I shift from JVM to JavaScript in one file.  Not sure how this would be known, but it would ROCK :slight_smile:

or

I have a Kara web appplication, and in my DSL for generating the HTML I reference a script LINK tag to my Kotlin-for-JavaScript build output file (“public/scripts/AppStuff.js”) and AppStuff.kt is within my same Kotlin Module (or another, but I really need them to result together into one tree for the running app, so module dependencies need to bring resulting scripts across to be used as well).

or

I have a Kara web appplication, and in my DSL for generating the HTML I reference a script LINK tag to my Kotlin-for-JavaScript file (maybe I say javaScriptFileForClass<MyJavaScriptThingy>) and it fixes up that link for me somehow so I don’t have a brittle relationship between resulting built .js filenames, although I guess with minimization and post processing maybe this is a bad idea.


#6

Going back to TypeScript, I know there is the ts2kt project working on converting the files.  What we need a bit more is something like this + webjars.  Or maybe the ability to wrap a webjar with the kotlin definitions?

So:

I use TypeScript .d.ts file for say Bootstrap, and use WebJars for Bootstrap.  So I create a JAR that contains the Typescript def and a manifest pointing at the WebJars instead of copying that myself.  No need to re-do their work.  So my Manifest just has a link to the other JAR (how? as a Gradle or Maven dependency this is possible to pull both)  Some way to relate and reuse?  Because for many libraries you have both these in existance already, why create something new for those that are there?

Then again, many people might do custom builds of some of the bigger libraries, but not always.

Just another thought on using what is out there.


#7

I see in this thread: https://devnet.jetbrains.com/message/5528828#5528828  that Ilya Ryzhenkov mentions multiple platform targeting/behavior, and that will apply to JS.  Maybe that affects some of my comments above?


#8

Stepping back a bit, here's how I imagine the ideal user story. A lot of this is obviously very far in the future, but in an ideal world for me, this is how things would go down.

I add a single dependency using whatever build tool is used by my project. That dependency happens to be com.handlebarsjs:kt-handlebars-2.0.0:1.0.0, the Kotlin bindings for Handlebars, because I need templating (the jar comes from Maven central, it’s binding version 1.0.0 for Handlebars version 2.0.0; this is probably not an optimal versioning scheme, but there should be one, clear, well-understood versioning scheme). Now IntelliJ and the compiler know exactly what API Handlebars provides.

Where to put the JS files? Should the build tool even do it? Depends! I think this is the tricky part.

  • Maybe I use bower to manage my JS libraries, and just need the type-safe bindings from the build tool. Achieving this should be a single line in my build file (something like `jsLibrarySystem "bower"`). If there's a mismatch between the version of the library in my bower.json and the Kotlin binding, I get a compile error. Maybe bower install is run as part of my build (if so, it should also be configurable). Maybe I don't even need to add the dependency, the build tool parses my bower.json and fetches the latest bindings for the libs I use.
  • Maybe I use some well-known web framework with a well-specified place where all JS files should be. I tell the build system that this is the case with just a line, possibly adding another line with the prefix for where I keep my JS library files relative to the root of my static assets. Duling the build process, the right JS files get placed in the right place.
  • Maybe I use some custom process; with just a few lines in my build file, the JS files are placed by the build process where-ever I want them to be.

EOF user story.

I agree that sharing code between JVM-JS is a very important feature; it’s already possible with build trickery, but ideally it should also be a well-supported case. To me, this is less important than being able to use JS libraries painlessly. It may make sense from a project management perspective as well: integrating two well-formed and stable products (the JVM and JS Kotlin toolchains) should  be simpler than integrating two half-formed tool-sets. Of course care must be taken not to make this impossible with decisions made before the integration.

I have just very vague ideas on how to best approach implementing this, so I won’t comment on that. Please let me know if I can clarify any of this, or if I can add more information :slight_smile:

One more word on the versioning scheme of the bindings: both the binding and the library can have breaking changes, so it would make sense to use semver for both. But of course only one of them will be recognized as an actual version-number by Maven and friends. I don’t have a deep knowledge of what’s available in this area, so again, I won’t make suggestions for the implementation, I’m sure you’ll come up with something that’s better than what I could make up.


#9

It's not yet in the master, I believe that will happen next week.

Stay tuned ;).


#10

Hi all,

Is any kind of dead code elimination on the road map?

i.e. It would be ideal if only the used poritions of libraries were included in the final JS output.

If so, I can’t see how it could be done if the sources were removed from the libraries in favour of binary descriptors and compiled JS.

Certainly the pre-compiled libraries would be the way to go for development, but an optimised build for deployment seems ideal.

Cheers,
Andrew


#11
What is the plan for sharing code between Kotlin JVM and JS?

This issue not about it. Anyway we think about it too, but I can't say any ETA, I hope it will be before 1.0 :) I created issue KT-6359, feel free to vote or star.

and also Maven/Gradle not being able to work.

If You mean kotlin2js project in Maven/Gradle inside IDEA it'll fixed soon(~next week). Thanks to Zoltan Nagy(aka Abesto) for the support kotlin2js in Gradle plugin.


#12

Feel free to create an issue about it in our tracker.

Thanks!


#13

- why not a foo.d.kt similar to typescript definition files?
We already have the similar feature with `native` annotations and we don't plan to fully drop them. Some reasons why we decide to use binary format: * compilation time -- we want to avoid analyze many definitions(from libraries) for each compilation. With d.ts like headers is complex, because we can't trust them. So we should reparse them and reanalyze at least headers. * Allow users to provide only headers when they don't want share sources (another possible solution is generate definition files, like d.ts). * We need the same data at runtime for support reflection. And maybe: * Posible to provide library as one js file.
- why not a typescript definition file?  (they have many available view DefinitelyTyped, no sense re-creating that world)

As you already said we are working on convertor typescript definition files to kotlin, it's the simplest way for us now(no need write own TS parser and typechacker), but maybe in the future we will able to work directly with d.ts.

- why not a dart external definition file? (they have the same problem, sucks to talk to JavaScript or other libraries with that barrier)

As far as I know whey solve the JS interoperability problems by dart:js library. And they used external definitions for internal things(like abstract something between DartVM and dart2js), I can't found other usages. Could you please share your knowledges about it? Thanks!

It would be really nice if dart, typescript, kotlin could interop with their definition files ... agree on something, not sure if binary or human readable.

Ideally Yes, but...

Also a binary format doesn't allow you to easily view what is available from the library.  So from "learnability" if I receive a library without original source, my IDE will need to be able to "de-compile" into something that looks like Kotlin to me so I can view it (similar to java class file decompilation).  Browsing via code-completion isn't always as productive as viewing the whole class quickly as text.

Good point. I hope we can provide deserialized view for user, anyway library owner may provide sources like for Java or our builtins.


#14

Thanks all for thoughts!


#15

I added a specific YouTrack on WebJars, I don't think copying *.js is complete enough:

https://youtrack.jetbrains.com/issue/KT-6463

It should copy everything under resources/webjars and you might need a path adjustment as well when copying.

Plus currently, even though the resources/webjars/**/*.js is copied, the compiler errors out due to the presence of the library and it not being the new kotlin jslib format.

Maybe it is a work in progress, but I think the **/*.js is the only planned items to copy, and that is wrong.  (images, css, html, templates, etc are all under that dir)