blazebit / blaze-expression Goto Github PK

The idea is, that we add a new parse tree visitor that extends the existing visitor. The subclass writes to a buffer that contains the refactored code. Every rule is extended to first do matching for refactorings, and if it matches, apply the rewrite to the buffer.

Given the code from the quick-start snippets:

DomainBuilder domainBuilder = Domain.getDefaultProvider().createDefaultBuilder();
domainBuilder.createBasicType("String", String.class).build();
//domainBuilder.withOperator("String", new DomainOperator[]{ DomainOperator.PLUS });
//domainBuilder.withPredicate("String", DomainPredicate.distinguishable());
domainBuilder.createEntityType("Cat")
    .addAttribute("name", "String")
    .build();
DomainModel domain = domainBuilder.build();

ExpressionService expressionService = Expressions.forModel(domain);
ExpressionCompiler compiler = expressionService.createCompiler();
ExpressionCompiler.Context context = compiler.createContext(
    Map.of("c", domain.getType("Cat"))
);
Expression predicate = compiler.createExpression("c.name = 'Hans'", context);

this throws:

Caused by: java.lang.UnsupportedOperationException: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with version 3 (expected 4).
        at org.antlr.v4.runtime.atn.ATNDeserializer.deserialize(ATNDeserializer.java:56)
        at org.antlr.v4.runtime.atn.ATNDeserializer.deserialize(ATNDeserializer.java:48)
        at com.blazebit.expression.impl.PredicateLexer.<clinit>(PredicateLexer.java:334)
        ... 22 more
Caused by: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with version 3 (expected 4).
        ... 25 more

I'd like to provide a full tutorial and example application with some out of the box integrations for a web-editor support based on a custom integration with the Monaco editor from Microsoft.

A few of the features(https://makecode.com/js/editor) that I'd like to adopt

• Syntax highlighting(improve coloring, formatting i.e. fat?)
• Auto suggestion of identifiers and keywords. Respect camel case such that s suggests clearScreen => The monaco editor does that already
• Function parameter info suggestion
• Function hover support
• Identifier information e.g. type name/function signature + documentation
• Put cursor into parenthesis when autocompleting a function
• Bracket matching
• Syntax validation
• Semantic validation(needs more type checking i.e. function arguments, operator resolvers shouldn't accept all types)
• Make use of Blazebit/blaze-domain#11
• Single-Line mode microsoft/monaco-editor#1601

It would also be cool if we could implement somekind of server side support through an LSP implementation via e.g. Xtext: https://www.eclipse.org/Xtext/documentation/330_web_support.html

Currently, the parsing fails completely when one argument uses the named invocation, but another doesn't. This should be more graceful i.e. we should hint that mixing named and positional invocation is disallowed.

The syntax highlighting in the function suggestion and hover box are non-optimal i.e. a question mark is treated as invalid token. We need to define a definition language that describes signatures and define a color scheme for that.

The definition language could look like this:

parseDefinition: definition EOF;
definition: functionDefinition | varDefinition;
varDefinition: typeDeclaration identifier;
functionDefinition: typeDeclaration functionName LP functionArguments RP;
functionArguments: (functionArgument (COMMA functionArgument QM?)* (COMMA functionVarArgument)?)?
functionArgument: typeDeclaration identifier;
functionVarArgument: typeDeclaration DOT DOT DOT identifier;
functionName: identifier;
typeDeclaration: identifier (LB identifier RB)?;

Add DATE and TIME literals to grammar to produce dates and times respectively. Also, allow comparison of date against timestamp and vice versa.

Our expression model is immutable, so it would be good if we allow configuring caching.

Add support for the bracket notation to filter collections. Within the brackets, an implicit variable _ exists (that can be omitted) to access the current collection element for the filtering. A possible example is priceHistory[price < 10 and date > TIMESTAMP 2020-01-01] is not empty. Collection filtering produces a collection again. The filtering should only be possible when the DomainOperator is enabled for the collection type. The interpretation implementation is straight forward. The persistence implementation requires help from the developer as that is dependent on the backing storage. We could try to attach support metadata on the collection types.

When #28 is done, we can introduce some special collection instance functions like the following:

• any - Returns any element
• min - Returns the minimum element, optional ordering argument?
• max - Returns the maximum element, optional ordering argument?
• size - Returns the size of the collection
• sum - Returns the sum of the collection elements, optional Function argument to extract value to sum?
• avg - Returns the average of the collection elements, optional Function argument to extract value to average?

Note though, that it would be nice if we could add the functions to specific collection types only i.e. use a receiver type of Collection[UserType] so that the function is only available for types that really support this.
This kind of also depends on #30

Expressions are sometimes allowed to produce multiple result types but when interpreting expressions it might be desirable to work only with a single type, so a type converter which is invoked on e.g. interpretAs(expression, BigDecimal.class) would be great.

The idea is to make the usecase of object filtering like #42 more efficient by implementing an engine that can split a predicate into parts that can be pushed down to act as filters for specific attributes.

A filter like root.name = 'abc' can be split into {"root": "name = 'abc'"}.

A filter like root.collection1.name = 'abc' could be split into {"root": "root.collection1 is not empty", "root.collection1": "name = 'abc'"}.

Maybe it makes sense to also allow empty collections i.e. apply left join semantics to such filters.

Some expression like e.g. 1.6 / 9.2 produce rational numbers which need to be rounded. Introduce a configuartion possibility to control the rounding and also introduce a way to switch to Long/Double types instead.

In a multi-line expression with the Monaco editor, the parameter hints are at wrong positions i.e. after the parameter argument.

When evaluating an expression for inlining, we don't render that through the literal rendering code and hence render the literal wrongly.

It would be great if expression serialization also works on subqueries or CTE-queries. At the moment it looks like that it's not going to work because the serializer expects an incompatible BaseQueryBuilder:

java.lang.ClassCastException: class com.blazebit.persistence.impl.FullSelectCTECriteriaBuilderImpl cannot be cast to class com.blazebit.persistence.BaseQueryBuilder (com.blazebit.persistence.impl.FullSelectCTECriteriaBuilderImpl and com.blazebit.persistence.BaseQueryBuilder are in unnamed module of loader 'app')
	at com.blazebit.expression.declarative.view.ManagedViewTypeCollection.add(ManagedViewTypeCollection.java:62)

if doing something like

cteBuilder = fullCriteraBuilder.fromSubquery(MyCte.class);
ExpressionSerializer<WhereBuilder> serializer = expressionService.createSerializer(WhereBuilder.class);
serializer.serializeTo(serializerContext, predicate, cteBuilder);

It would be awesome if we could implement an API that allows to filter a list of objects and their collections based on a filter predicate. I think that the Blaze-Expression language is restricted enough so that we can implement that in general.

I think that we will need to implement this by first translating the object graph to a tuple list so that we have a tuple per collection element, filter the tuple list by the predicate and finally try to filter the object graph based on the resulting tuple list. Maybe we don't need the tuple list translation step, but there are definitely challenges to solve for predicates that involve multiple collections like e.g. root.collection1.name = root.collection2.name.

To implement an interpreter cache for expressions we need to capture the function volatility. I propose to add the following two categories:

• VOLATILE - Function return can change at any time
• IMMUTABLE - Function returns the same result for the same arguments

The cache should be stored in the interpreter or interpreter context and thus does NOT depend on #18 as it should be unbounded. It should be possible to disable the interpreter cache though.

I would have expected the translation of an expression to fail if no persistence metadata can be found, but a user reported the translation worked and produced a wrong result. We should aim to throw an exception instead.

It's sometimes necessary to access certain state in SPIs like e.g. ExpressionRenderer. We should provide access to registered services through a context there and in other SPIs.

Due to e.g. database column constraints, it makes sense to allow specifying a maximum length.

IntelliJ has it and I really like it. Seems people requested this for the monaco editor already, but it should be doable by using decorations: microsoft/vscode#16221 (comment)

To support user defined text templates that can make use of the expression language, we should introduce a custom syntax to embedd the expression language in arbitrary text. I will use #{ and } for this purpose, where pre-text #{expression} post-text logically translates to 'pre-text ' + expression + ' post-text' + escaping of special tokens in the text.

Support for this should be added through a dedicated API that can parse that. A serializer that emits this template form should also be available. The monaco editor should also support this with special highlighting of the delimiters #{ and } and auto-completion support only within the delimiters.

The first version should be able to translate expressions to Formulas that appear in the same row.

Currently we don't suggest argument names for named invocations, but we should.

Like IntelliJ, I think we should suggest for each argument paramN = .

Not sure how important this is, but for bigger expressions it might make sense to use some kind of let expression syntax to avoid duplication/improve readability. I propose a syntax like the following:

let a = 1 + 1,
    b = a / 10,
    c = 20
in
    a * c

Implementation-wise, the use of let expressions will make no difference. I even propose that we implement this as parse time expansion.

It would be cool if we could also suggest keywords in e.g. the monaco editor. It should be doable to identify what tokens are expected next. We should obviously only suggest word-based tokens, no punctuation.

Also look into how https://github.com/mike-lischke/antlr4-c3 could help here or maybe even just use that library.

When a root for a path is not found, a ImplicitRootProvider should be invoked to see if that can infer some root for a path. This should be registered in the ExpressionCompilerContext.

It should be possible to use the instance method syntax for functions that support this (requires Blazebit/blaze-domain#7).

Most of the time, the expected result type of an expression is known. Add the possibility to pass a list of expected result types to the createEditor function and validate against that.

When a type is used as function argument a type adapter is applied twice, leading to an error.

There are some use cases in the collection functions area where it would be beneficial if a user could provide some kind of user defined function for extracting values. I'm proposing to add a Function type for this purpose that has two type parameters, input and output type. Collection functions could then use that type as argument type. Syntax-wise I would allow the user to provide a custom name for the input i.e. the following: priceHistory.sum(entry -> entry.price). Not sure yet if a short-hand syntax is a good idea, but maybe we could allow priceHistory.sum(#price).

Add support for the common comment syntax // for single line comments and /* */ for multi-line comments.
Comments should NOT be retained in the expression AST.

The idea is to create a new expression visitor that generates an anonymous class, implementing the whole expression logic in a single method. This should probably depend on a shaded version of ASM and live in a dedicated module.

Not sure when or if this will be done at all, but I wanted to at least record the idea.

Seems we messed up the precedence for some operators since * has higher precedence than / although it should have the same and be parsed from left to right.

Serializing an expression (a + b) * c results in a + b * c i.e. the serializer is not respecting the operator precedence. Let's also check if the excel serializer is affected by that.

Since functions can return entity domain types, the monaco editor integration should support auto-suggestion for functions as well. Currently we only support auto suggestion for atoms and paths.