I/O Streams for Kotlin

I would like to start from thanking all Kotlin creators! :slight_smile:

I have read “Kotlin in Action” book and it best book about programming that I read recently!
Thanks a lot for authors of this book!

Since this summer I become a big fan of Kotlin!
I see no reason continue writing on Java in new projects now (at least in internal projects).

I learning Kotlin in writing code and feel pleasure in comparing with Java and C# (and sure enough C++ :).

I have created I/O library on Kotlin for my personal reasons, but I think it can be useful for other Kotlin users. Especially Kotlin hasn’t own I/O library at moment.

I have published my library named the ‘binary-streams’ on BitBucket and Maven Central.

Summary of the binary-streams:
Author: Alexander Kornilov
Version: 0.1
License: Apache v2.0
Project URL: https://bitbucket.org/akornilov/binary-streams
API documentation: https://akornilov.bitbucket.io/doc/binary-streams
Design document (very light :): https://bitbucket.org/akornilov/binary-streams/downloads/binary-streams.pdf
Gradle dependency on Maven Central: compile ‘org.bitbucket.akornilov.kotlin:binary-streams:0.1’
Maven Central: https://repo.maven.apache.org/maven2/org/bitbucket/akornilov/kotlin/binary-streams/0.1/

Main features:

  • Extended support of data streams with configurable byte-order.
  • The size of an integer in bytes is arbitrary for data streams.
  • Integers with a size larger than 64-bit are supported using the BigInteger .
  • Possibility to choose signed or unsigned representation of an integer.
  • Unsigned integer (64-bits and more) using the BigInteger .
  • Float and Double with specified byte order.
  • Hint about fetch ability.
  • Read or write an arbitrary number of bits from the BitStream .
  • Can seek to a specified bit in the BitStream .
  • Build-in support of string encoding: ASCII, UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE.
  • Detection and writing BOM for text files.
  • Reading chars as code points at all.
  • Reads all lines from a text file in " .useLines " Kotlin style.
  • String encoding and byte order might be set for all stream or specified in place.
  • Input or output buffering on the fly with a specified buffer size.
  • Stream implementation for random access file with reading and writing buffering.
  • Stream implementation to read or write from kotlin.ByteArray .
  • Adapters for java.io.InputStream and java.io.OutputStream .
  • Helpful tools to make easy work with bits, code-points and other binary operations in StreamUtils.kt .

Classes diagram (from design document):

Code examples (from design document):
// Creates stream over java.io.InputStream with default stream byte order big-endian and default buffer size 4096.
FileInputStream(“file.bin”).toBinaryBufferedStream(byteOrder = ByteOrder.BigEndian).use { stream →

// Reads byte as unsigned integer
stream.readByteUnsigned()

// Reads Double with specified in place byte-order
stream.readDouble(ByteOrder.LittleEndian)

// Skips ten bytes
stream.skip(10)

// Reads BigInteger with size 10 bytes and unsigned representation
stream.readLong(10, false)

// Can read three bytes?
stream.canRead(3)

// Reads UTF-16 string contains 20 code points
stream.readString(StringEncoding.UTF16, 20)

}

// Creates stream over java.io.OutputStream with native byte order and default string encoding UTF-8
FileOutputStream(“file.bin”).toBinaryStream().use { stream →

// Writes single byte
stream.writeByte(33)

// Writes long value using += operator
val longValue = 0xFE30023L
stream += longValue

// Flushes data
stream.flush()

// Write code point 65 in UTF32 encoding
stream.writeChar(65, StringEncoding.UTF32)

// Writes BOM in current stream encoding
stream.writeBom()

// Writes line in current stream encoding
stream.writeLine("Hello")

// Writes double value
stream.writeDouble(33.3)

}

// Creates stream for random access file for read/write and default read buffer 4096:
File(“file.bin”).openBinaryStream(false).use { stream →

// Try detect BOM
if (stream.tryDetectBom()) {
    // The BOM was detected and default stream encoding and byte order were updated.
}

// Seeks to last 20 bytes of the file.
stream.position = stream.size - 20L

// Reads Int value from the stream
val intValue: Int = stream.read()

// Writes Int value
stream.write(intValue)

// Seeks to begin of the file
stream.position = 0

// Reads signed integer with size 3 bytes.
stream.readInt(3, true)

// Reads all lines to the end of file
stream.forLines {
    for (line in it) {
        println(line)
    }
}

}

// Reads all lines from file in UTF-32
FileInputStream(“file_urf32.txt”).toBinaryBufferedStream(encoding = StringEncoding.UTF32).useLines {
for (line in it) {
println(line)
}
}

// Opens BitStream and read/write bits.
BitStream(File(“file.bin”).openBinaryStream(false)).use { stream →

// Seeks to 99 byte
stream.position = 99

// Seeks to 3rd bit in 99 byte
stream.offset = 3

// Seeks to 451 bit
stream.bitPosition = 451L

// Reads byte
stream.readByte()

// Read bit
stream.readBit()

// Read 33 bits as signed integer
stream.readBits(33, true)

// Write bit 1
stream += true

// Write bit 0
stream += false

// Write 4 bits
stream.write(0b1101, 4)

// Read 128 bits to BigInteger as unsigned
stream.readBigInteger(128, false)

}

// Creates stream over ByteArray
val byteStream = StreamByteArea(ByteArea(16))

// Reads Short
byteStream.readShort()

// Seek in stream
byteStream.position = 10

2 Likes

The BitStream is exactly what I need to serialize/de-serialize a low-level binary format incorporating Binary trees. I had written a BitStream implementation some years ago in Java, but I will not re-invent the wheel (or at least re-port the wheel) in Kotlin where it looks like you have crafted a fine one already :slight_smile: . Thank you for the good work, I am looking forward to using this.

Thank you! If you’ll need assist with BitStream library go ahead and ask me.

1 Like

The project was moved to another hosting:
Project URL: https://kotlin-utils.sourceforge.io
API documentation: binary-streams
Design document (very light :): Download kotlin-utils from SourceForge.net
Gradle dependency on Maven Central: compile ‘net.sf.loggersoft.kotlin:binary-streams:0.33’
Maven Central: Central Repository: net/sf/loggersoft/kotlin/binary-streams

Hi [akornilov82], while I assume no obligation on your part, to maintain binary-streams; I thought I would let you know that I’ve just tried to recompile my project that relied on binary-streams:0.33. Possibly something broke in Kotlin binary compatibility during these 3 years of development (Kotlin 1.3.311.5.20) because at runtime I now get:

'long loggersoft.kotlin.streams.BitStream.readUBits-s-VKNKU(int)'
java.lang.NoSuchMethodError: 'long loggersoft.kotlin.streams.BitStream.readUBits-s-VKNKU(int)'

…while executing expression: readUBits(bits = someBits).toInt()

Hello.

Please try binary-streams v0.35 compiled for Kotlin 1.5.20 and let me know if all fine :slight_smile:

1 Like

Hi @akornilov82 . Thank you! That resolved the problem, my utility App runs again.

Hi @akornilov82.
First of all thank you for this library. It’s indeed the missing bit when compared with the kotlinx-io library.

With that said, I think I’ve an issue with the latest version.
In particular, I’m using a BitStream and I get the bytes written in little endian instead of big ending.
For example, the number 66 gets written as 66 00 instead of 00 66.
Moreover, the defaultByteOrder doesn’t seem to be considered at all.

val os: OutputStream = // get it somehow
val obs = BitStream(os.toBinaryStream(ASCII, BigEndian))
val length: UShort = 66u
obs.writeUShort(length)

fun BitStream.writeUShort(value: UShort) = write(value.toULong(), UShort.SIZE_BITS)

Hi again @akornilov82.
After some deeper debugging, I realized you’ve the nativeOrder hardcoded in the BitStream class

private inline fun writeCache(bytes: Int) = stream.writeInt(cache, bytes, nativeByteOrder)

and this resolved to little endian on my machine.
Could you please avoid to hardcode such value and rely instead on the defaultByteOrder of the underlying stream?

Hi again @akornilov82.
I’ve forked - for test purposes - and removed the nativeByteOrder from the method call, but I’ve now realized the situation is even more broken than expected.
Since you handle everything either as a long, when you perform the actual write you write an entire long in BE/LE which is the cumulation of the bytes received so far, but this is wrong.
From an API point of view, I should be able to write as BE/LE a single data type and not their cumulation to long.

For example (in BE):
Given:

val a: UShort = 1u
val b: UShort = 2u

obs.writeUShort(a)
obs.writeUShort(b)

Expected result:

[0, 1, 0, 2]

Result got:

[0, 2, 0, 1]

Hello,

Thank you for your notice!
I understood the issue, fix will come soon.

Hi @akornilov82.
Thanks for your rapid reply.

While you’re on it, bumping Kotlin to the latest version and removing the need to opt-in unsigned types (they’re not experimental anymore) would be nice.
Also, although it’s a pure Kotlin project, a module-info definition would not be bad to have, so that for JDKs >= 9 things which should not be accessible are nicely hidden (not exported).

Hello,

Sorry for delay. Please take a look to the new version of binary-stream v0.37:
https://repo.maven.apache.org/maven2/net/sf/loggersoft/kotlin/binary-streams/0.37/
https://kotlin-utils.sourceforge.io/doc/binary-streams/binary-streams/loggersoft.kotlin.streams/index.html

Library has been update to Kotlin 1.7.10 and now BitStream implements Stream interface. So, now you don’t need functions like this ‘fun BitStream.writeUShort(value: UShort) = write(value.toULong(), UShort.SIZE_BITS)’.
Just use BitStream as Stream.

Just curious, where is the working repository located?

Hello. To use library just add those lines to your build.gradle script (Groovy version):

repositories {
    mavenCentral()
}
dependencies {
    implementation 'net.sf.loggersoft.kotlin:binary-streams:0.37'
}

I think the question was about the source code. I guess it is here: kotlin-utils / binary-streams / [9b5238]

1 Like

Hi @akornilov82

Am I wrong, or the issue with the BE/LE ordering is not fixed yet?
In the code of v0.37, after a rapid look, I can still see that the native byte ordering is used when writing the cache content instead of the one specified during the construction of the stream.

Please see the issue described in I/O Streams for Kotlin - #10 by cdprete

Hello,

No, the issue has been solved.
In your examples please just remove function-extension fun BitStream.writeUShort(value: UShort) = write(value.toULong(), UShort.SIZE_BITS) and try use writeUShort of BitStream itself. Now such writeXXX and readXXX (for all primitive types) methods avaliable because BitStream now implements Stream interface. You can choose byte order in place (by use byteOrder parameter of fun writeShort(value: Short, byteOrder: ByteOrder = defaultByteOrder) ) or change default byte order for whole stream.
The cache of BitStream is step below layer with byte order. It should be passed in forward order in any case.
Please feel free to ask me if something not clear or BitStream works not as expected.

I’ll give it a try.
Thanks for your efforts.

Is there a new PDF out there which shows how to use the new version?

@akornilov82 am I wrong or UByte is missing?
I can see it’s possible to read UShort, UInt, ULong but not UByte yet.