arrow-kt / arrow-meta Goto Github PK

Currently, quote transformations are limited to Transform.replace which replaces a returned node by a list of declarations. There is an additional set of operations that we would like transformation to be used on.

We would like to support further transformations:

We should do one PR per transform and ensure these transformations are tested with the quoteOutputMatchescombinators in the testing library. The testing library would need to receive a few additional algebras to verify produced sources in the newSources transform

We need to implement a separate site for the Arrow Meta library. Once done this, we will implement the same way with the other Arrow libraries.

This is a list of known Services we can hook into - by investigating the compilers StorageManager. The goal is to aggregate them and simplify, when and at which Phase we can mutate Services:
This list is highly unstable and will change frequently.
These are the known ExtensionPhases - here declared using the Arrow Meta DSL:

- Config
- PackageProvider
- [This is Phase is going to be added: TypeResolution/ TypeResolver]
- AnalysisHandler
- SyntheticResolver
- DeclarationAttributeAlterer
- StorageComponentContainer
- ClassBuilder
- Codegen
- IRGeneration
- SyntheticScopeProvider
- DiagnosticsSuppressor
- PreprocessedVirtualFileFactory

#Singleton:
Solely means that the compiler requires a Singleton Service.
It does not accept multiple Services with this Interface.
And we need to figure out how to do this efficiently with JavaReflection

In contrast, services without the #Singleton can be registered for free.

The following services are structured by their packages - currently all in TypeResolution:

resolve

descriptors

types

context

builtins

incremental

serialization

kotlin.reflect

config

load

This issue proposes the implementation of KtDestructuringDeclaration quote and scope. it's related to #89

@type typealias PositiveInt = Int with Positive<Int>


val x: PositiveInt = -1 //fails
val y: Validated<Not<Positive>, Positive<Int>> = PositiveInt(maybePositive)

@type typealias Option<A> = Either<Unit, A> with Monad<EitherPartialOf<Unit>>

Option is automatically materialized as a new type which combines the runtime of Either alongside its Monad API as synthetic injected methods

This issue proposes the implementation of KtFinallySection quote and scope. it's related to #89

With the compiler plugin we can automatically track purity in non-suspend functions and error if the user is performing unsafe ops and not inside suspend or IO.
The steps for this implementation may look like:

• Intercept with ClassBuilderInterceptorExtension new methods and function declarations
• Error for all non-suspended functions returning Unit.
• Error for all non-suspended functions returning any type whose inner body expressions are type checked as returning Unit.
• Error for all non-suspended functions that use known side-effecting 3rd party APIs that are blacklisted outside of suspend.
• Provide error messages for all cases where possible returning the shape of the correct implementation or include detailed instructions on how to fix them.

The current code base contains a partial implementation of some of these ideas that can serve as inspiration until we figure the arrow-meta biz.

https://github.com/47deg/arrow-meta-prototype/blob/0cd8b6449ed8950d6df4f58f7707dd6dc5e732e2/compiler-plugin/src/main/java/arrow/plugin/TestClassBuilderInterceptorExtension.kt#L28

With type classes and the use of KEEP-87 arrow-kt/arrow-meta-prototype#2 with compiler plugins, it should be possible to:

• Derive an HList representation for all data classes
• Derive a Coproduct representation for all sum types
• Derive automatic inductive instances for HList and Coproduct
• Given the above-mentioned derivation features, we could then use the isomorphism between data class with HList and sealed classes with Coproduct to derive automatic type class implementation for data classes and sealed classes for any given type class

This issue proposes the implementation of KtTryExpression quote and scope. it's related to #89

Currently highlighting in the Editor is not automatically updated once the meta cache is rebuilt in computeCache.

If one scroll or moves around highlighting eventually works and shows the generated descriptors but we would like this to happen as soon as the meta cache is populated.

Additionally the MetaPlugin it's bootstrapped as part of the PackageFragmentProvider and synced so it only initializes once. Ideally, we would find an earlier event in which we can bootstrap the meta system.

The configuration to publish Arrow Meta artifacts (snapshots and releases) in Bintray is being added in #40.

However, Gradle Plugin is not included in #40 because it has its own way to be published to the Plugin Portal and also its own version roadmap:

https://plugins.gradle.org/docs/publish-plugin

The desired name for the Gradle plugin would be:

io.arrow-kt.arrow

We have to create 4 icons for Arrow Meta:

1. Nothing: ⊥
2. Polymorphic: ∀
3. Bind: >>=
4. Injection: *

We want a base DSL based on how ScalaMeta quasiquotes work, so we can generate files using string interpolation.

All the plugins for Arrow will rely on it for any imaginable features, so it's a very blocker requirement.

This issue is to keep track of the line of thought that I followed in the implementation of KEEP-87. Every step I took will be linked to one or more commits containing the implementation of the ideas outlined that, hopefully, will be easily relatable to how things are done with the compiler plugin.

These are the steps I took to implement KEEP-87.

1. Add Type classes feature to the language (link)

In order to be able to support the new feature and enable the new syntax, I had to create a language feature. The new keywords that are introduced are linked to this feature and I'd assume they are only enabled if this feature is selected. This is necessary in the case of experimental features, as can be seen in the other declarations in the same file.

Plugin version: I believe this won't be necessary in the compiler plugin, as we are not modifying the language and the application of the plugin is optional.

2. Add new keywords with and extension (link)

New tokens for the keywords with and extension were added. This automatically adds highlighting in IntelliJ IDEA.

Plugin version: As this will be implemented with annotations @with and @extension, we may not need to do anything.

3. Define applicability of with and extension (link)

I had to specify which elements can be modified with the keywords with and extension, namely value parameters (with) and classes/objects (extension).

Plugin version: similar to the previous point, since these keywords are added as annotations, I assume you have already constrained their applicability in the annotation definition.

4. Add flags for with and extension (link)

Internal representations for ClassDescriptor and ValueParameterDescriptor (together with a big cascade of changes related to different implementations of these interfaces) needed to be modified to encode the presence of the modifiers extension and with.

Plugin version: as we have the possibility to inspect which annotations are present during compile time, I guess this is not needed if that information is carried into ClassDescriptor and ValueParameterDescriptor (which, if I remember correctly, is present).

5. Disable reporting of unused values / classes (link)

Implicit arguments and type class instances are not explicitly used and therefore are marked with a warning as unused. These warnings are silenced.

Plugin version: I would say this could be out of the scope of this plugin, but should be possible to mark something as used if it is resolved by the resolution algorithm. I'd say this has lower priority than other tasks.

6. Modify the resolution algorithm to include implicit parameters in scope

Consider the following example:

interface Semigroup<A> {
  fun combine(x: A, y: A): A

  companion object {
    extension object IntSemigroup: Semigroup<Int> {
      override func combine(x: Int, y: Int): Int = x + y
    }
  }
}

fun combine(x: Int, y: Int): Int = x * y

fun dup(x: Int, with semigroup: Semigroup<Int>): Int = combine(x, x)
fun dup2(x: Int, semigroup: Semigroup<Int>): Int = combine(x, x)
fun dup3(x: Int, semigroup: Semigroup<Int>): Int = semigroup.combine(x, x)

val first = dup(5) // Should be 10
val second = dup(5, Semigroup.IntSemigroup) // Should be 10
val third = dup2(5, Semigroup.IntSemigroup) // Should be 25
val forth = dup3(5, Semigroup.IntSemigroup) // Should be 10

Calls need to be resolved in a different manner if there are implicit parameters in the scope:

• In dup, the function call to combine must be the one defined in Semigroup, shadowing the combine function defined at package level.
• In dup2 we have a semigroup parameter, but it is not marked as with, so the resolved combine is the one at package level.
• In dup3 we are making explicit reference to the combine function from Semigroup.

What I did in this case was to simulate the presence of with(parameter) blocks wrapping up the function body, but being careful to respect the scope of this.

Plugin version: I modified BodyResolver, but not sure how it can be done in the case of the plugin.

7. Generate bytecode to use implicit parameters in function body (link)

Once that implicit parameters are correctly added to the scope, they need to be properly loaded in the stack for the function body.

Plugin version: not sure how the plugin API handles this.

8. Resolve and instantiate implicit parameters in function calls (link)

In an early version the resolution algorithm to find a candidate instance for a given parameter was done in the code generation stage. The algorithm had to be moved to an early stage of the compilation process in order to be able to report errors properly.

The description of this algorithm can be read in the text of the PR submitted to JetBrains.

Once candidates are uniquely resolved, they need to be instantiated in their corresponding position of the function call:

• For extension object, there is a single instance of them that is loaded.
• For extension class, their constructor needs to be invoked with the parameters they require. Most likely those parameters will be implicit as well, so the algorithm for resolution has to consider those cases.

Plugin version: for the first part, the resolution algorithm should be very similar to the one that is already implemented, as it is based on *Descriptors. For the second part, it is up to the possibilities offered to generated bytecode.

9. Other generalizations to the resolution algorithm and error reporting

• Orphan instances in subpackages
• Using implicits in closures / inlined lambdas
• Resolving concrete instances (e.g. fun combine(x: Int, with IntSemigroup))
• Resolving looking into the type / type class hierarchy

With compiler plugins is very likely we can implement full-blown type classes including resolution and automatic derivation.

Type class derivation

• By default look first for user declared extensions following the KEEP-87 candidate resolution.
• (User) Denote type class instances as derivable by annotating the annotation method flagged as @derivable. For example in Functor<F> @Derivable Kind<F, A>.map...`.
• Intercept declaration and class building phase to scan all interface which contains a single type argument. These are the type class candidates.
• Intercept declaration and class building phase of all types in search for compatible matches that can be used to implement the @derivable method if it exists. If the type class contains no derivable method we stop otherwise we try to find a compatible match.
  In this step we also want to skip types that already include instances for this type class following the KEEP 87 resolution rules.
• When a matching type is found for a derivable method we can create automatically a type class instance for it and enrich the data type companion with the typeclass instances as a property.
• This can potentially be done in the ClassBuilderInterceptorExtension phase.

Call site resolution

• For all expressions that make a call into a type class constrained function look for instances that may be injected with the same resolution rules at KEEP-87

This feature should be part or is closely related with arrow-kt/arrow#1638

This issue proposes the implementation of KtIsExpression quote and scope. it's related to #89

This issue proposes the implementation of KtReturnExpression quote and scope. it's related to #89

This issue proposes the implementation of KtWhileExpression quote and its scope mapping. this issue is related to #89

This issue proposes the implementation of KtWhenExpression quote and scope. it's related to #89

Currently, we are unable to test icon providers and other IDE specific features in Arrow Meta.
The purpose of this task is to come up with an internal framework that includes the arrow meta IDE plugin and tests plugins using it's DSL work in the Editor.

This should allow us to add such tests in a CI environment.

Explain how the dependency tree of the compiler-plugin, gradle-plugin and idea-plugin works and how a user would get it.

• The goal is to provide a set-up that only downloads the most minimal binaries for a working set-up.

Hence a CI/CD pipeline only needs the gradle and compiler plugin, whereas a Developer and alike needs IDEA support.

This issue proposes the implementation of KtTryExpression quote and scope. it's related to #89

This issue proposes the implementation of KtThrowExpression quote and scope. it's related to #89

We can leverage compiler plugins to automatically rewrite imperative flows that use Kind<F, A>.bind() into rewrites that are simply flatMap based. This will free us from the stack labels machinery which currently uses reflection, ties us to the JVM and it's slow.

An implementation may do the following:

• Find all declarations of fx blocks
• Digg into the function ordered KtCallExpression and find all suspend calls that use !
• If there are dependencies between the order of operations Rewrite the function body so each ordered ! it translates into a flatMap chain where the remaining expressions below are nested ending in a final map.

fx {
  val a = !fa
  val c = !fb(a)
  c + 1
}

becomes:

fa.flatMap { a -> 
  fb(a) 
}.map { c ->
  c + 1
}

This is similar as to how scala for comprehensions work and it would free us from being dependent on suspension and in general the coroutine limitations around fx

We need to come up with a list of the entire set of AST elements and declarations that we want the user to be able to destructure with templates. Right now, we only have scopes for templates for classOrObject, func, property and type aliases.

The sole purpose of scopes is to provide template destructuring capabilities and all methods in their API should be related to rendering properties but not conditional logic or other type of operations that belong in the PSI and that we don’t have to worry about maintaining.

Every element of the KtElement hierarchy is worth restructuring with a template, which needs a Scope type. We want to make it easy to build nodes for the AST as quotes. Ideally, we should have one file per quote and that includes the public function and its scope.

List of AST elements completed

• KtBinaryOperation #90
• KtForExpression #120
• KtLoopExpression #111
• KtWhileExpression #121
• KtLambdaExpression #122
• KtFunctionLiteral #123
• KtWhenEntry #139
• KtWhenCondition #145
• KtFinallySection #156
• KtBlockExpression #154
• KtCatchClause #152
• KtIfExpression #135
• KtWhenExpression #136
• KtTryExpression #151
• KtFinallySection #156
• KtThrowExpression #162
• KtIsExpression #164
• KtReturnExpression #165
• KtAnnotatedExpression #167
• KtTypeElement #127
• KtDestructuringDeclaration #176
• KtValueArgument #183
• KtParameterList #188
• KtTypeReference #132
• KtObjectDeclaration #133
• KtImportDirective #182
• KtModifierList #186
• KtDotQualifiedExpression #298
• KtThisExpression #325
• KtClassBody #326
• KtPackageDirective #336
• KtPropertyDelegate
• KtScript
• KtSimpleNameExpression
• KtEnumEntry
• KtDeclaration
• KtCallableReferenceExpression
• KtPrimaryConstructor
• KtStringTemplateExpression
• KtConstructorDelegationCall
• KtSuperTypeEntry
• KtSuperTypeCallEntry
• KtInitializerList

TODO Scopes for AST Elements

• KtTypeElement
• KtFile
• KtBodyExpression
• KtFunctionLiteral
• KtDestructuringDeclaration
• KtTypeReference
• KtModifierList
• KtParameterList

Creating a PR for AST elements

There's a lot of elements, so we're in need of a system to keep track of all these PRs:

1. Link your PR to the corresponding issue.
2. We'll add the AST element you're writing and link the corresponding issue in the List of Elements section
3. Label every PR & Issue with AST-scope:

### Checklist for [element]
 - [ ] Added/replaced the inverse element in ElementScope
 - [ ] Destructure and make available all methods related to rendering properties, but no logic
 - [ ] Add documentation for element, including an example using all properties
 - [ ] Testing added for validation to ensure all properties can be used as a commutative identity

As an addition to the GradleProjectImportHandler there should also be an AbstractMavenImportHandler instance for the compiler plugin. The implementation is pretty similar to the existing Gradle version. This issue is just a reminder moving forward.

Here is an example: https://github.com/JetBrains/kotlin/blob/65244b4bea81f737466618927d4f3afe339cad0d/plugins/kotlin-serialization/kotlin-serialization-ide/src/org/jetbrains/kotlinx/serialization/idea/KotlinSerializationMavenImportHandler.kt#L12

This issue proposes the implementation of KtIfExpression quote and scope. it's related to #89

This issue proposes the implementation of KtBlockExpression quote and scope. it's related to #89

We're building out the quote system with all as many AST elements as we can, but we will need help to go back to explain examples of what each AST element is in their custom scopes.

Potential helpful sources:

• the corresponding PsiElements Kotlin compiler code - see com.kotlin.psi package.
• the PSIViewer IntelliJ plugin can be helpful for discovering PSI elements and their associated class:

• Kotlin Academy is sometimes helpful, especially in their segment for Kotlin Programmer Dictionary. For example, one blog covers Function Type vs Function literal vs Lambda expression vs Anonymous function

In addition to the compiler dsl the ide dsl provides ide plugin dev's and alike to write their plugin features alongside their compiler plugin or completely write ide plugins with the dsl.
The DSL changes the way ide plugins are written and let users describe features in terms of functions instead of classes.

Users can manipulate the Editor Environment explicitly, without the need for class creation.

Add the Traversable Functors API to scope and it's elements. This will allow us to fold and transform scopes in templates with a known API.

This issue proposes the implementation of KtWhenCondition scope. it's related to #89

The original version of the purity plugin checks when a function returns Unit or contains calls in its body to other functions that it return Unit.

This plugin should show visually in the editor this is happening and provide intentions to refactor the function to become suspend. This is the IDE piece of the purity plugin.

With a compiler plugin we can easily change what the irCall for == and others operators are dispatched to. == is currently a fun called EQEQ

This means we can replace the call and if there are instances of Eq and Order and use them instead of the default functions that delegate to the kotlin internals or equals impl.
This will decouple the lang from the inheritance hierarchy on the ordering and equality operators and would allow us to use == instead of eqv since we can dispatch == to a type class instance instead of the data type member for equals.

CALL 'public final fun EQEQ (arg0: kotlin.Any?, arg1: kotlin.Any?): kotlin.Boolean declared in kotlin.internal.ir' type=kotlin.Boolean origin=EQEQ

That can be intercepted and replaced by a new invocation that delegates to the instance of the type class first and if not to the EQEQ call. Same interception strategy the KEEP-87 plugin uses to enhance function access value parameters.

We should rewrite the support using the DSL we must create in arrow-kt/arrow-meta-prototype#11

This is a proposal issue for Ide features we might want to consider for Arrow.

Comment below for suggestions and Feedback.

Synthetic resolution based on computeCache only happens when the files are compiled and in .class form in the local module output.

This is inconvenient because users get redlines and no auto-completion help until they compile.

We would like to be able to run Meta there automatically as the user is typing on the editor buffer to offer the same SynthDescriptors we are offering now but before it compiles the full project.

Additionally, @jereksel pointed out an extension in the IDE we are not using and apparently apparently may help us solve some of these problems. We need to investigate it and see how we can use it. org.jetbrains.kotlin.idea.core.extension.KotlinIndicesHelperExtension

I have the feeling that this extension may help us inject automatically functions and declarations which are callable descriptors to contribute to their scope as the user types. Running the quote system on each keystroke may not be expensive if we do something like:

1. Grab the KtFile we are on and feed to a quote
2. The quote then already traverses the PSI tree and matches it
3. The quote internally already generates a new KtFile
4. We use the resolve API to partially type check the new KtFile as it is done now in the computeCache function linked above and add it to the computed cache before compiling. This generates the synth descriptors which are fed to the PackageFragmentProvider and SynthResolverExtension automatically since these are invoked as the user types and the user should see the new synth descriptors visible in the editor.

As the Test's for the existing LM is about to be merged in #129, we may discuss what Text or GoTo actions we want to implement for Meta, wether Users click on the Icon, which then directs them on their default browser to Doc's and tutorials or display the Text.

Interestingly enough, LineMarkers have in most cases short messages. If we do want to present more such as we do in the PolyLineMarker or much more, I'll work on API's which opens up TextDisplays on the right-hand side with Content we want to communicate, similar to the Toolbar on the right-hand side for Gradle or the Markdown ide plugin. The same API could also be used for the AnkIdePlugin to render the Docs.
cc @raulraja @franciscodr @nomisRev

The current docs website uses JavaScript to set the active class for a menu entry if it's the current one.

When rendering a page, this makes the menu to be shown collapsed for half a second, and then when this JS function kicks in, the menu opens and the associated entry gets highlighted applying the CSS related to that active class.

This can be avoided by applying this class to the menu entries on the website at build time, instead of runtime through JavaScript. An example of this can be seen at the nef website:

https://github.com/bow-swift/nef/blob/master/docs/_includes/_sidebar.html

https://nef.bow-swift.io/docs/

This is an easy ticket for first contributors in Meta.
The remaining LineMarkers in Arrow need Tests. One example is here
https://github.com/arrow-kt/arrow-meta/blob/master/idea-plugin/src/test/kotlin/arrow/meta/ide/plugins/higherkinds/HigherKindTest.kt using the Testing DSL.

TODO:

• Optics LM
• Poly LM
• Comprehensions LM

We currently facing a few issues during registration:

• Multiple Registrations in MetaPluginRegistrarComponent

metaPlugin.registerMetaComponents(project, configuration)

As of today we're applying for each Project window the IdeMetaPlugin, thus Extensions, which are applied on Application level will have duplicates.

• Undisposed Extensions
  The Disposable in ExtensionProvider needs to be disposed, same applies for MetaPluginRegistrarComponent#disposeComponent. If not, all registered Extensions are not removed from memory, which leads to memory leaks and higher RAM consumptions. Possible solutions encompase to lift MetaPluginRegistrarComponent to an ApplicationComponent and define a Disposable strategy without the need of the PicoContainer, but this is an open discussion.

• Make Extensions dumbAware
  We'll need to indicate Extensions, which are available during Indexing and supply that in the IdeSyntax. DumbAware and DumbService should facilitate us with sufficient tools.

These need to be dealt with after we have a sufficient test environment and before we release the Alpha. The first 2 have High Priority, where as the last is a minor fix we can do after the release. Feel free to correct me @jansorg.
We are open for suggestions.

We should change the Jekyll version up to 4.0.0 since we can get the benefit of its improvements, also we need to change sidebar links to relative_url in order to navigate in the proper different versions of Arrow meta.

This issue proposes the implementation of KtLoopExpression scope, this scope will permits us to apply a ast quote mapping over loop KtExpressions such KtForExpression and KtWhileExpression. this issue is related to #89

This issue proposes the implementation of KtAnnotatedExpression quote and scope. it's related to #89

This issue proposes the implementation of KtModifierList quote and scope. it's related to #89

Hi team!
I created this issue to keep in mind all the material we have to develop for Kotlin's presentations.
Feel free to edit this issue to add more information, dates, etc..
@ahinchman1 @rachelcarmena

Kotlin Days: London

Deadline for presentation design: Friday 22 November (?) TBD
Meeting date: Thursday 8 November at ??:?? TBD
Slides

KotlinConf: Copenhagen

Deadline for presentation design:
Meeting date:

JFuture Belarus

Deadline for presentation design:
Meeting date:

Thanks!

This issue proposes the implementation of KtForExpression quote and its scope mapping, it depends on #111 and should wait for KtLoopExpression scope implementation before starts with this issue. this issue is related to #89

This issue proposes the implementation of KtWhenEntry scope.. this is related to #89

Hello Arrow Team,

I started a new AST-Parsing Library kotlinx.ast (https://github.com/kotlinx/ast) that can be used to read Kotlin source files into easy to use data classes (using antlr-kotlin and the official Kotlin language grammar).

I'm already using this library for code generation, in combination with kotlinpoet (https://github.com/square/kotlinpoet) to create the generated files.

Maybe it can be used in arrow-meta. Please let me know your requirements for code generation if you are interested in discussing this.

kotlinx.ast is currently JVM-only, but support for JavaScript and Native is planned. It should be easy to create a shared code generation module for arrow-meta that can be used inside a gradle plugin, in unit tests and even in a native code generation tool (using kotlin native).

@rachelcarmena , @raulraja: A starting point for discussing may be replacing kastree with kotlinx.ast (https://github.com/arrow-kt/arrow/tree/rr-meta-prototype-integration/modules/meta/arrow-meta-prototype/compiler-plugin/src/main/kotlin/kastree/ast)