Mikrator (a Liquibase wrapper): changelogs as code, easy usage

1

I always wanted to integrate a database migration tool, like Liquibase, into projects. But I didn’t quite like the untyped changelog files or some APIs. I love Kotlin DSLs and static typing, so I had the idea of creating a wrapper that supplies both.

Here is the first PoC alpha version of that: GitHub - Batix/mikrator: A Liquibase wrapper for easy usage and changelogs as code.

To save you a click, I reproduce the example code below:

val config = Config {
  sqlShowAtLogLevel = Level.INFO
}

val changeLog = ChangeLog {
  changeSet("1-create", "John Doe") {
    changes {
      createTable("tbl") {
        column("id", "INT")
      }
    }
  }

  changeSet("2-insert", "John Doe") {
    changes {
      insert("tbl") {
        column("id") { numericValue(1) }
      }

      tagDatabase("v1")
    }
  }

  changeSet("3-insert", "John Doe") {
    changes {
      insert("tbl") {
        column("id") { numericValue(2) }
      }
    }

    rollback {
      delete("tbl", where = "id = 2")
    }
  }
}

Mikrator(openDatabase(sqlConn), config).use { mikrator ->
  mikrator.update(changeLog)

  if (weNeedToGoBack && mikrator.tagExists("v1")) {
    mikrator.rollback(changeLog, "v1")
  }

  logger.trace { "db snapshot:\n${mikrator.snapshot().serialize(FileFormat.YAML)}" }
}

What do you think? How do you do database migrations?

2 Likes
2

Looks good, but I’d like to see some type safety around specifying column types, rather than just writing “INT”.

Also, it’d be super cool if the DSL could somehow track the structure of the table, and prevent you from inserting the wrong data type. IE, in the second change set, you have numericValue(1). It’d be awesome if the DSL could know whether numericValue is valid for that column in that table. That said, I know this would probably be a pretty big feature, and also probably kind of replicates what the database (and maybe Liquibase) does already.

1 Like
3

Thanks! Codifying the possible column types would be cool, but it’s not feasible, because those are pretty dynamic. E.g. you have different available column types for different database types / servers. You can run changesets for specific database types with the dbms parameter. So the possible column types you can choose depend on the database type the changelog is run against and the specified dbms parameter (which can also be someting like “mssql,oracle,mysql,!mariadb”). I guess there is even more complexity to it.

The *Value methods (or the Liquibase internals) do some auto-converting between types. So you always have a set of choices, e.g. it would also be valid to use stringValue("1") or booleanValue(true) here.

Regarding tracking table structure: tables evolve over time, that’s what a migration tool is for. :slight_smile: Tracking the structure is not a good fit here, and also not feasible (you would need to track that table A, even in the same changeset, is A_1 after createTable and A_2 after addColumn etc.).

The changelog records changes, not complete, wanted states. You can do something like “sync the structure of my entity classes to the database” with Exposed’s SchemaUtils or MigrationUtils, but those have their limits (that’s why I wanted to try out a migration tool).

A Kotlin tool with those features would be cool, but I feel that would need to be created from the ground up, this is just a wrapper around Liquibase - with the implication that you need to read up on Liquibase stuff, but the benefit of having a nicer way of specifying things and using it.

4

Thing is, in your original post, you said this:

I agree. XML/JSON is just text, and while you can have IDE plugins that can validate your Liquibase migration files for you, it’s even better if you have compile-time type checking. But what you’ve created is a static structure that mimics the XML/JSON structure, but still leaves the data types completely untyped. IMO, I’m much more likely to get a specific data type wrong than I am to get “createTable” or “changeSet” wrong.

I get that different databases have different data types. However, there’s a lot of common types, and I would say more common types than there are different types.

I would propose two changes. Firstly, create a defined set of the common types as an enum that can be passed to the column function. Second, potentially create one enum per database type that contains all the unique data types for that specific database, or just leave the column function that accepts a String as the second argument. That way, for common types that are shared across most/all databases, you get type safety, but then you get the flexibility for custom types specific to certain databases, by passing in a String.

5

The main things about “untyped changelog files” I wanted to get rid of were questions like “what changes are available? with what parameters?” and the need for file-system / classpath-resources access (via ResourceAccessor). Maybe there are schemas available for the Liquibase JSON / YAML format to solve the first, I don’t know, but those also have their limits. So I made this. :slight_smile:

Regarding the typing of columns: An enum alone wouldn’t be enough, as types can have parameters, like varchar(100) or INT UNSIGNED (for specific databases only), so either something with sealed classes or methods with parameters (varcharColumn(100)) would be needed. Probably methods, as those are more easily discoverable.

Aside: Kinda off-topic, but somehow IntelliJ does a poor autocompletion job lately, or is it just me? Try setting objectQuotingStrategy in ChangeLog(). It does not display the enum or its entries. :man_shrugging:

Regarding the typing of values: Just to surface some complexity, there are different contexts where column() {} can be used:

  • createTable: inside column() {} (with many parameters like autoIncrement) there’s constraints()
    • and default {} (in which you can call the *Value methods)
  • addColumn: the same (with some other parameters like afterColumn)
    • additionally there’s value {}, also with *Value methods (including some additional ones for current sequence value and blob / clob)
  • insert (for VALUES), delete (for WHERE): here column() {} has no parameters (except name)
    • the *Value methods are available directly inside column() {}
  • update (two times: for SET and for WHERE)
    • same

Also, I don’t really know what the common types across all databases are (a subset of those?). Liquibase Data Type Handling can be a starting point, but leaves much to be desired (many empty cells). What would we do, if a database is added, that doesn’t support a (up to that point) common type? Probably just keep it as-is.

Btw, getting the data type wrong is not that bad, because of the auto-conversion (either from Liquibase or the database).

I see the appeal and am always in favor of more static typing, but I feel all this is a bit too dynamic. :confused: That doesn’t mean I’m completely against it, maybe someone comes up with a nice solution.