Hey all,
We have a use case at work where we deal with tables that contain more than 20+ fields. The compilation time starts to grow very quickly for each field added and at a certain point - compilation time takes more than an hour which is extremely poor ergonomics for our users especially with multiple tables containing lots of fields.
I have narrowed the problem down to the way insertion is implemented via BindableBuilder
which causes implicit re-resolution of each element that is part of the HList in the implicit chain each time a field is added causing this massive overhead. So for example, if you insert a String, you'll end up with String :: HNil and if you then insert an Int then you'll end up with
Int :: String :: HNil where the implicit resolution of (String :: HNil) is not cached and needs to be resolved once again resulting in a very slow process since now you need to resolve String :: HNil and then Int :: String :: HNil. If you had five fields, it would end up like this:
f1 :: f2 :: f3 :: f4 :: f5 :: HNil
requires
f5 :: HNil
5 times
f4 :: f5 :: HNil
4 times
f3 :: f4 :: f5 :: HNil
3 times
f2 :: f3 :: f4 :: f5 :: HNil
2 times
f1 :: f2 :: f3 :: f4 :: f5 :: HNil
1 time
I have found that we don't even end up using the types that we are tracking for inserts and that it's needed purely for Shapeless' applyProduct. I have also looked into the way Doobie does this as I wanted to compare if Doobie has the same limitation but it appears that Doobie has a very different way of formulating their string interpolator and they don't have this limitation.
I have managed to work around this limitation by creating a simple builder for Insert, Updates (in-progress) and Selects (in-progress) which do not rely on applyProduct
and by association BindableBuilder
which brings compilation time back down to a matter of milliseconds and avoids rebuilding implicit re-resolution chains as much as possible. Here's a little snippet:
package com.ringcentral.cassandra4io.dsl
import cats.Monad
import cats.syntax.flatMap._
import cats.syntax.functor._
import com.datastax.oss.driver.api.core.cql.BoundStatement
import com.ringcentral.cassandra4io.CassandraSession
import com.ringcentral.cassandra4io.cql.Binder
private[dsl] class Insert[State <: InsertState](
private val keyspace: Option[String],
private val table: String,
private val columnNames: Vector[String],
private val columnValueSetter: (BoundStatement, Insert2.Index) => (BoundStatement, Insert2.Index)
) { self =>
def column[A](columnName: String, columnValue: A)(implicit ev: Binder[A]): Insert2[InsertState.NonEmpty] = {
val updatedSetter: (BoundStatement, Insert2.Index) => (BoundStatement, Insert2.Index) =
(boundStatement, index) => {
val (bsUpdated, indexUpdated) = columnValueSetter(boundStatement, index)
val (bsResult, indexResult) = ev.bind(bsUpdated, indexUpdated, columnValue)
(bsResult, indexResult)
}
new Insert[InsertState.NonEmpty](
keyspace = self.keyspace,
table = self.table,
columnNames = self.columnNames :+ columnName,
columnValueSetter = updatedSetter
)
}
def render(implicit ev: State =:= InsertState.NonEmpty): String = {
val fullyQualifiedTableName =
keyspace
.map(k => s"$k.$table")
.getOrElse(table)
val columns = columnNames.mkString(start = "(", sep = ", ", end = ")")
val values = columnNames.indices.map(_ => "?").mkString(start = "(", sep = ", ", end = ")")
s"INSERT INTO $fullyQualifiedTableName $columns VALUES $values"
}
def execute[F[+_]: Monad](session: CassandraSession[F])(implicit ev: State =:= InsertState.NonEmpty): F[Boolean] = {
val query = render
session.prepare(query).flatMap { preparedStatement =>
val initial = preparedStatement.bind()
val (result, _) = columnValueSetter(initial, 0)
session
.execute(result)
.map(_.wasApplied)
}
}
}
object Insert {
type Index = Int
def apply(table: String, keyspace: Option[String] = None): Insert2[InsertState.Empty] =
new Insert2[InsertState.Empty](keyspace, table, Vector.empty, (bs, i) => (bs, i))
}
/** Phantom type used to prevent an insert statement from being built without data */
sealed trait InsertState
object InsertState {
sealed trait Empty extends InsertState
sealed trait NonEmpty extends InsertState
}
Usage:
val insert =
Insert("test_data_multiple_keys")
.column("id1", 1L)
.column("id2", 1)
.column("data", "lol")
.execute(session)
This is just as type-safe as the cql string interpolator (and not much better in terms of type safety) but it provides our users with a much nicer alternative to slow compilation times. I would love to improve the string interpolated version but that would require a lot of time. Would you guys consider taking in MRs for these builders in the mean time?