We are using the Kotlin scripting capabilities to store input and output data combined with their processing script. To transform stored input data to output data we load the stored script into a Kotlin scripting engine, evaluate it and call a specific function on it that actually applies the transformation/business logic.
Since the first evaluation by the engine is quite expensive regarding execution time we decided to introduce a caching layer. This layer shall enable us to reuse any script until it changes and has to be recompiled.
Basically this works fine so far. However, we ran into some problems regarding the caching/reusing aspect. We tried two approaches of implementations for this.
- In the first one we tried to cache CompiledScript-instances using a single scripting engine instance.
- In the second one we cached entire engine instances for each script version.
The first attempt failed since the application slows down and finally gets stuck after some re-compilations/-evaluations.
The second attempt basically works but runs into a memory leak using the standard scripting engines Kotlin provides in its libraries. We are aware of comments in the script engine code indicating that there is still some cleanup of resources required.
However, we could only get rid of this memory leak by introducing the possibility to do some kind of cleanup by introducing a function for our custom scripting engine that shuts down the compiling service daemon:
daemon.scheduleShutdown(false)
Although we can confirm that this tweak seems to solve the memory leak problem within the bounds of our project, we are not sure if it is this approach is appropriate. Besides that we are overall unsure if this is the intended use of Kotlin scripting functionality (in terms of best practice) and/or if we might introduce other side effects by shutting down scripting daemons.
Overall we do like the possibility to embed the feature of dynamically evaluated Kotlin code in our application(s) and would like to take it further but we are unsure if we are on the right track regarding caching of scripts/engines.
We are interested in comments and opinions by experienced users or developers of the kotlin scripting environment regarding the use of the scripting engine and the memory leak issue. Moreover, we are interested in the reason for failure of the first variant, in which we reuse the same engine but cache the CompiledScripts instead. If one looks at the documentation of the general Java scripting environment it seems that caching CompiledScripts is the “best practice” way to avoid unnecessary recompilations.
In order to illustrate the problem and our approach(es) we attached a small example (Maven) project that contains three test methods, each of them producing one of the described behaviors (1., 2. with and 2. without memory leak). Maybe this helps to comprehend our textual description.
kscriptingmemoryleak.tar.gz (5.4 KB)
thanks
Ben