Teleproto using chimney macros

[urmaul]

This is a placeholder for discussions about migrating teleproto to use chimney macros inside.

WIP draft implementation for this can be found here: moia-oss/teleproto#338

[MateuszKubuszok]

Soo, this PR still relies on using Transformers (and maybe PartialTransformers) as the basis.

What I had in mind is more like this:

package io.scalaland.chimney.internal.compiletime

import io.scalaland.chimney.internal.compiletime.{datatypes, ChimneyDefinitionsPlatform}

trait ChimneyAsLibrary
    extends Derivation
    with ChimneyDefinitionsPlatform
    with datatypes.IterableOrArraysPlatform
    with datatypes.ProductTypesPlatform
    with datatypes.SealedHierarchiesPlatform
    with datatypes.ValueClassesPlatform
    with rules.TransformImplicitRuleModule
    with rules.TransformSubtypesRuleModule
    with rules.TransformToSingletonRuleModule
    with rules.TransformOptionToOptionRuleModule
    with rules.TransformPartialOptionToNonOptionRuleModule
    with rules.TransformToOptionRuleModule
    with rules.TransformValueClassToValueClassRuleModule
    with rules.TransformValueClassToTypeRuleModule
    with rules.TransformTypeToValueClassRuleModule
    with rules.TransformEitherToEitherRuleModule
    with rules.TransformMapToMapRuleModule
    with rules.TransformIterableToIterableRuleModule
    with rules.TransformProductToProductRuleModule
    with rules.TransformSealedHierarchyToSealedHierarchyRuleModule

package io.moia.protos.teleproto.internal

import io.scalaland.chimney.internal.compiletime.ChimneyAsLibrary

import scala.reflect.macros.blackbox

class EncoderMacros(val c: blackbox.Context)
    extends io.scalaland.chimney.internal.compiletime.ChimneyAsLibrary
    with ... /* teleprotos own rules e.g. EncodeImplicitRule */ {

  def deriveEncoderWithDefaults[
      From: WeakTypeTag,
      To: WeakTypeTag
  ]: Expr[Encoder[From, To]] = {
    // pseudocode:

    val inputName = freshTermName(...)
    val inputExpr: Expr[From] = c.Expr[From](q"$inputName")

    val cfg = TransformerConfiguration() // customize, read config with DSL etc
    val context = TransformationContext.ForTotal(inputExpr, cfg)
    
    deriveFinalTransformationResultExpr(context).toEither.fold(
      derivationErrors => /* report somehow */,
      toExpr =>
        c.Expr[Encoder[From, To](
          q"""
          new Encoder[${Type[From]}, ${Type[To]}] {
            def encode($inputName: ${Type[From]}): ${Type[To]} = $toExpr
          }
          """
        )
    )
  }
  
  // chain of responsibility of sort: attempt first rule, it it yields another, then another, etc
  final override protected val rulesAvailableForPlatform: List[Rule] = List(
    // TransformImplicitRule, // looks for Transformer/PartialTransformer, instead provide your own:
    EncodeImplicitRule, // <- this should look for implicit Encoder
    TransformSubtypesRule,
    TransformToSingletonRule,
    TransformOptionToOptionRule,
    TransformPartialOptionToNonOptionRule,
    TransformToOptionRule,
    TransformValueClassToValueClassRule,
    TransformValueClassToTypeRule,
    TransformTypeToValueClassRule,
    TransformEitherToEitherRule,
    TransformMapToMapRule,
    TransformIterableToIterableRule,
    TransformProductToProductRule,
    TransformSealedHierarchyToSealedHierarchyRule
  )
}

Something like this would be able to:

• remove dependency on Transformer and PartialTransformer type classes
• allow injecting custom rules
• remove the need of vampyre macros or similar
• customizing the error message

[MateuszKubuszok]

It was just a quick pseudocode but I wanted to give a general idea.

[urmaul]

Thank you, I'll look into it. My idea was to copypaste the outermost macro and then iteratively dig deeper while keeping it compiling and working all the time. That's probably more work but less big changes and opportunities to get lost.

[MateuszKubuszok]

Basically Chimney at this point has a customizable derivation engine of sort:

• config, input, overrides are passed around with TransformerContext
• derivation Rule make a chain of responsibility where the first match creates result, rule can also fail or yield
• DerivationResult is like Either (of non empty list of failures) + Writer (logs)
• whole thing take src: Expr[From] and returns Expr[To] or Expr[partial.Result[To]] wrapping with a type class is done in the entrypoint to the macro

So I would see teleproto creating the context on its own, reusing some rules and creating a few on its own (e.g. summoning implicit Encoder/Decoder) and having its own entrypoint calling the engine and wrapping its output.

[MateuszKubuszok]

I've updated the draft a bit:

src/main/scala/io/scalaland/chimney/internal/compiletime/ExportChimneyDerivationEngineForTeleproto.scala - part of the macro that can be shared between Scala 2 and Scala 3, exposing Chimney's parts that are relevant to derivation (with a name that would prevents conflicts)

package io.scalaland.chimney.internal.compiletime

trait ExportChimneyDerivationEngineForTeleproto
    extends derivation.transformer.Derivation
    with derivation.transformer.Configurations
    with derivation.transformer.Contexts
    with derivation.transformer.ImplicitSummoning
    with derivation.transformer.ResultOps
    with datatypes.IterableOrArrays
    with datatypes.ProductTypes
    with datatypes.SealedHierarchies
    with datatypes.SingletonTypes
    with datatypes.ValueClasses
    with derivation.transformer.integrations.OptionalValues
    with derivation.transformer.integrations.PartiallyBuildIterables
    with derivation.transformer.integrations.TotallyBuildIterables
    with derivation.transformer.integrations.TotallyOrPartiallyBuildIterables
    with derivation.transformer.rules.TransformationRules

src/main/scala-2/io/scalaland/chimney/internal/compiletime/ExportChimneyDerivationEngineForTeleproto.scala - Scala 2-specific implementations of abstract interfaces, using Scala 2 macros:

package io.scalaland.chimney.internal.compiletime

trait ExportChimneyDerivationEngineForTeleprotoPlatform
  extends ExportChimneyDerivationEngineForTeleproto
    with ChimneyDefinitionsPlatform
    with datatypes.IterableOrArraysPlatform
    with datatypes.ProductTypesPlatform
    with datatypes.SealedHierarchiesPlatform
    with datatypes.ValueClassesPlatform
    with derivation.transformer.rules.TransformImplicitRuleModule
    with derivation.transformer.rules.TransformSubtypesRuleModule
    with derivation.transformer.rules.TransformToSingletonRuleModule
    with derivation.transformer.rules.TransformOptionToOptionRuleModule
    with derivation.transformer.rules.TransformPartialOptionToNonOptionRuleModule
    with derivation.transformer.rules.TransformToOptionRuleModule
    with derivation.transformer.rules.TransformValueClassToValueClassRuleModule
    with derivation.transformer.rules.TransformValueClassToTypeRuleModule
    with derivation.transformer.rules.TransformTypeToValueClassRuleModule
    with derivation.transformer.rules.TransformEitherToEitherRuleModule
    with derivation.transformer.rules.TransformMapToMapRuleModule
    with derivation.transformer.rules.TransformIterableToIterableRuleModule
    with derivation.transformer.rules.TransformProductToProductRuleModule
    with derivation.transformer.rules.TransformSealedHierarchyToSealedHierarchyRuleModule

src/main/scala-3/io/scalaland/chimney/internal/compiletime/ExportChimneyDerivationEngineForTeleproto.scala - Scala 2-specific implementations of abstract interfaces, using Scala 3 macros:

package io.scalaland.chimney.internal.compiletime

// using q: Quotes, require us to make it abstract class on Scala 3 to make it a stable identifier
abstract class ExportChimneyDerivationEngineForTeleprotoPlatform(q: scala.quoted.Quote)
  extends ChimneyDefinitionsPlatform(q)
    with ExportChimneyDerivationEngineForTeleproto
    with datatypes.IterableOrArraysPlatform
    with datatypes.ProductTypesPlatform
    with datatypes.SealedHierarchiesPlatform
    with datatypes.ValueClassesPlatform
    with derivation.transformer.rules.TransformImplicitRuleModule
    with derivation.transformer.rules.TransformSubtypesRuleModule
    with derivation.transformer.rules.TransformToSingletonRuleModule
    with derivation.transformer.rules.TransformOptionToOptionRuleModule
    with derivation.transformer.rules.TransformPartialOptionToNonOptionRuleModule
    with derivation.transformer.rules.TransformToOptionRuleModule
    with derivation.transformer.rules.TransformValueClassToValueClassRuleModule
    with derivation.transformer.rules.TransformValueClassToTypeRuleModule
    with derivation.transformer.rules.TransformTypeToValueClassRuleModule
    with derivation.transformer.rules.TransformEitherToEitherRuleModule
    with derivation.transformer.rules.TransformMapToMapRuleModule
    with derivation.transformer.rules.TransformIterableToIterableRuleModule
    with derivation.transformer.rules.TransformProductToProductRuleModule
    with derivation.transformer.rules.TransformSealedHierarchyToSealedHierarchyRuleModule

Then in teleproto:

src/main/scala/io/moia/protos/teleproto/internal/TeleprotoDefinitions.scala - use it as this: TeleprotoDefinitions => when defining your own cross-compilable mixins:

package io.moia.protos.teleproto.internal

trait TeleprotoDefinitions
  extends io.scalaland.chimney.internal.compiletime.ExportChimneyDerivationEngineForTeleproto
  // ...
  {

  // definitions of teleproto types e.g. Encoder, Decoder

  // interfaces for e.g. summoning implicit Encoder, Decoder

  // code which would parse teleproto configs and turn them into TransformationContext for Chimney
}

src/main/scala-2/io/moia/protos/teleproto/internal/TeleprotoDefinitionsPlatform.scala:

package io.moia.protos.teleproto.internal

trait TeleprotoDefinitionsPlatform
  extends TeleprotoDefinitions
  with io.scalaland.chimney.internal.compiletime.ExportChimneyDerivationEngineForTeleprotoPlatform {

  // implementations of teleproto types e.g. Encoder, Decoder

  // implementations for e.g. summoning implicit Encoder, Decoder
}

package io.moia.protos.teleproto.internal

import scala.reflect.macros.blackbox

class TeleprotoMacros(val c: blackbox.Context)
    extends TeleprotoDefinitionsPlatform
    with ... /* teleprotos own rules e.g. EncodeImplicitRule */ {

  def deriveEncoderWithDefaults[
      From: WeakTypeTag,
      To: WeakTypeTag
  ]: Expr[Encoder[From, To]] = {
    // pseudocode:

    val inputName = freshTermName(...)
    val inputExpr: Expr[From] = c.Expr[From](q"$inputName")

    val cfg = TransformerConfiguration() // customize, read config with DSL etc
    val context = TransformationContext.ForTotal(inputExpr, cfg)
    
    deriveFinalTransformationResultExpr(context).toEither.fold(
      derivationErrors => /* report somehow */,
      toExpr =>
        c.Expr[Encoder[From, To](
          q"""
          new Encoder[${Type[From]}, ${Type[To]}] {
            def encode($inputName: ${Type[From]}): ${Type[To]} = $toExpr
          }
          """
        )
    )
  }

  def deriveEncoderWithDefaults[
      From: WeakTypeTag,
      To: WeakTypeTag
  ]: Expr[Encoder[From, To]] = {
    // pseudocode:

    val inputName = freshTermName(...)
    val inputExpr: Expr[From] = c.Expr[From](q"$inputName")

    val cfg = TransformerConfiguration() // customize, read config with DSL etc
    val context = TransformationContext.ForPartial(inputExpr, cfg)
    
    deriveFinalTransformationResultExpr(context).toEither.fold(
      derivationErrors => /* report somehow */,
      toExpr =>
        c.Expr[Encoder[From, To](
          q"""
          new Encoder[${Type[From]}, ${Type[To]}] {
            def encode($inputName: ${Type[From]}): ${Type[To]} = ${convertPartialToPbResult(toExpr)}
          }
          """
        )
    )
  }
  
  // chain of responsibility of sort: attempt first rule, it it yields another, then another, etc
  final override protected val rulesAvailableForPlatform: List[Rule] = List(
    // TransformImplicitRule, // looks for Transformer/PartialTransformer, instead provide your own:
    EncodeImplicitRule, // <- this should look for implicit Encoder
    TransformSubtypesRule,
    TransformToSingletonRule,
    TransformOptionToOptionRule,
    TransformPartialOptionToNonOptionRule,
    TransformToOptionRule,
    TransformValueClassToValueClassRule,
    TransformValueClassToTypeRule,
    TransformTypeToValueClassRule,
    TransformEitherToEitherRule,
    TransformMapToMapRule,
    TransformIterableToIterableRule,
    TransformProductToProductRule,
    TransformSealedHierarchyToSealedHierarchyRule
  )
}

To understand this architecture:

• basically whole derivation logic is macro agnostic
• one is defining interfaces (including Expr, Type) and methods operating on them and whole engine only uses these types and methods
• we add the implementation at the end of the world with mix-ins (so the whole macro is a giant cake pattern, kinda unavoidable with path-dependent types)
• the limitation is that deriveFinalTransformationResultExpr(context) can only derive Expr[To] (total) or Expr[partial.Result[To]]. The former works for Encoders, but Decoders would have to convert PbResult into partial.Result within the macro (e.g. converting result of a decoding with implicit Decoder), and from partial.Result to PbResult at the exit from the macro
• it's a bit more complicated than it needs to be since, it's the first time someone reuses the engine, and a lot of interfaces are package-private to not pollute the intellisense. If this approach works out, we could expose these traits in some module I guess?

[MateuszKubuszok]

I drafted this PR as an example how Chimney can be use in another library with different type classes, and how to make it cross-compilable between Scala 2 and Scala 3.

If this approach makes sense, we could release chimney-engine as an additional module intended for other libraries to use, but the way we implement cross-compilation kinda forces the architecture on any adopter.

[MateuszKubuszok]

There is no such utility right now. Enums in particular are returned as a ListMap when using SealedHierarchies utility, and they have their name as a key of the list, but no way of obtaining such name OOTB.

I made a #600 for this. In some other project where I used chimney-macro-commons I temporarily worked around this limitation by:

    def simpleName[A: Type]: String = {
      val colored = Type.prettyPrint[A]
      val mono = "\u001b\\[([0-9]+)m".r.replaceAllIn(colored, "")
      val start = mono.lastIndexOf(".") + 1
      val end = mono.indexOf("[", start) - 1
      mono.substring(start.max(0), if (end < 0) mono.length else end)
    }

which isn't optimal, but for a while it might be okayish.

[urmaul]

Oh great, thank you!

[MateuszKubuszok]

I added some docs, and merged #574 - chimney-engine should get published on Sonatype snapshots, but as I wrote in the added docs - this module should not be considered stable, nor its version a semantic version.

I don't expect many breaking changes, but some changes be necessary in the TransformationContext to unblock several features in the future. Maybe I'll find a way to not make them breaking, maybe not. 😅 It should not happen soon as I don't have much capacity for OSS lately, and I don't think recent merges would be published as a release, unless more people would contribute improvements in my stead.

[MateuszKubuszok]

Derivation-engine got released in 1.5.0 - as I wrote in the docs and changelog, the API might change (I think I might need to sometimes change something in the TransformationContext to implement the remaining feature requests from the backlog) but it shouldn't happen often.

[urmaul]

FYI the infinite implicit call loop is fixed now. I've added ctx.config.isImplicitSummoningPreventedFor[From, To] check in our custom implicit summoner rule.

[urmaul]

FYI I dropped the attempts when it turned out there's no easy way to handle forward- and backward-compatibility annotations the way teleproto did it. Also it turned out that using chimney directly for our stuff is much easier than a year ago so migrating our apps to use chimney is eatier than migrating teleproto.

Maybe I'll extract the prefixed proto enum migration but not sure if it's worth it.

Thank you for the help.