protostuff / protostuff-compiler Goto Github PK

There are two proto files in the same module that has a message with same name and package.

Expected result
Generator should detect this issue and throw an error.

Actual result
Generator appends generated code into a single file.

Generated java code is broken if there is a field result of any type except int32.

https://developers.google.com/protocol-buffers/docs/reference/proto3-spec#service_definition

Service definition

service = "service" serviceName "{" { option | rpc | stream | emptyStatement } "}"
rpc = "rpc" rpcName "(" [ "stream" ] messageType ")" "returns" "(" [ "stream" ]
messageType ")" (( "{" {option | emptyStatement } "}" ) | ";")

Example:

service SearchService {
  rpc Search (SearchRequest) returns (SearchResponse);
}

A field named class will generate invalid Java code - the corresponding getClass() method is already defined in the Object superclass.

The following error is reported when compiling:

error: getClass() in XXX cannot override getClass() in Object

protoc generates following java code:

/**
 * <code>optional int32 class = 1;</code>
 */
int getClass_();

Source: protostuff/protostuff#163

I have message with map field:
message PairsMsg {
...
map<string, string> pairs = 1;
...
}
and got this error
ERROR i.p.c.c.ProtostuffCompilerCli - Unknown option: 'map_entry' [unknown]

Oneof-s are already supported by parser, this task is about adding support for oneof to HTML and Java generators.

Generate proto files from proto files, apply different transformations as a part of the process.

Rename ANTLR rules - they should conform to https://developers.google.com/protocol-buffers/docs/reference/proto2-spec

Maps should be serialized as a key-values according to google protobuf specification.

Currently they are serialized as a repeated list of messages.

Parser must be fully compatible with proto3.

why does the generator not map the proto type int32(int64) to java type Integer(Long)?

Is there any sample for how to use the parser and then go over the object model?

io.protostuff.Message<T> can provide a method like T self() ?

// auto generated by compiler
class A implements io.protostuff.Message<A>{
  public A self(){
     return this;
  }
}

then user case

public static <T> byte[] ser(io.protostuff.Message<T> message) {
  if (message == null) return new byte[] {};
  final Schema<T> schema = message.cachedSchema();
  byte[] data;
  try {
    data = ProtobufIOUtil.toByteArray(message.self(), schema, BUFFER.get());
  } finally {
    BUFFER.get().clear();
  }
  return data;
}

when i generate code from proto file with grpc service, it got error. does it not support grpc service yet?
proto file
service PBBasicService { rpc callApi(PBMessageRequest) returns(PBMessageResponse); }
logs
[ERROR] Caused by: org.stringtemplate.v4.misc.STNoSuchPropertyException: no such property: io.protostuff.compiler.model.ServiceMethod.javaReturnTypeFullName

Implement java code generator (basic) to be used with existing protostuff api/core runtime.

Given proto descriptor for service:

service TestService {

    rpc test(RequestMessage) returns(ResponseMessage);
}

Current implementation of java code generator produces following code:

@javax.annotation.Generated("io/protostuff/it/service.proto")
@Service("io.protostuff.it.TestService")
public interface TestService {

    @Rpc
    @Request("testRequest")
    @Response("testResponse")
    CompletableFuture<ResponseMessage> test(RequestMessage request);

}

There is no way to change generated code.

Following things should be customizable:

1. Future type: we can keep CompletableFuture as a default, but there should be a way to use different one, for example - Guava's ListenableFuture.

TODO: more customizations?

Log messages are too long, would be nice to make them shorter:

[INFO] Write C:\Users\konstantinsh\IdeaProjects\server\game\game-table-api\target/generated-sources/proto\com/playtech/live/tableservice/internal/FetchEventsRequest.java
[INFO] Write C:\Users\konstantinsh\IdeaProjects\server\game\game-table-api\target/generated-sources/proto\com/playtech/live/tableservice/internal/FetchEventsResponse.java

For example, skip ${project.basedir} part.

Requirements: maven plugin + build from source should work in maven 3.0.
Documentation: Maven Plugin

Although still one of the most popular Markdown parsing libraries for the JVM, pegdown has reached its end of life.

The project is essentially unmaintained with tickets piling up and crucial bugs not being fixed.

pegdown's parsing performance isn't great. In some cases of pathological input runtime can even become exponential,
which means that the parser either appears to "hang" completely or abort processing after a time-out.

Therefore pegdown is not recommended anymore for use in new projects requiring a markdown parser.

Instead I suggest you turn to @vsch's flexmark-java, which appears to be an excellent replacement for these reasons:

• Modern parser architecture (based on commonmark-java), designed from the ground up as a pegdown replacement and
  supporting all its features and extensions
• 30x better average parsing performance without pathological input cases
• Configuration options for a multitude of markdown dialects (CommonMark, pegdown, MultiMarkdown, kramdown and Markdown.pl)
• Actively maintained and used as the basis of an IntelliJ plugin with almost 2M downloads per year
• The author (@vsch) has actively contributed to pegdown maintenance in the last two years and is intimately familiar with pegdown's internals and quirks.

In case you need support with migrating from pegdown to flexmark-java, @vsch welcomes inquiries in here or here.

Maven version 3.5.0 is giving the following warning:

[WARNING] *****************************************************************
[WARNING] * Your build is requesting parallel execution, but project      *
[WARNING] * contains the following plugin(s) that have goals not marked   *
[WARNING] * as @threadSafe to support parallel building.                  *
[WARNING] * While this /may/ work fine, please look for plugin updates    *
[WARNING] * and/or request plugins be made thread-safe.                   *
[WARNING] * If reporting an issue, report it against the plugin in        *
[WARNING] * question, not against maven-core                              *
[WARNING] *****************************************************************
[WARNING] The following plugins are not marked @threadSafe in Live UMS Network Gateway API:
[WARNING] io.protostuff:protostuff-maven-plugin:2.0.0-alpha33
[WARNING] Enable debug to see more precisely which goals are not marked @threadSafe.
[WARNING] *****************************************************************

When using protostuff maven plugin.
It seems that the project is missing the @threadSafe annotation.

Hi,

when adding the json_name option to a proto file (see https://developers.google.com/protocol-buffers/docs/proto3 -> JSON Mapping -> notes on 'message' in the table), we are getting the following parser exception using protostuff-compiler 2.2.27:

io.protostuff.compiler.parser.ParserException: Unknown option: 'json_name' [test.proto:25]
at io.protostuff.compiler.parser.OptionsPostProcessor.processStandardOptions(OptionsPostProcessor.java:145)
[...]

Is this a known issue, or do we need to handle this case differently?

Regards,
Stefan

Parser should be compatible with proto2.
Full compatibility is not required.

This task is mainly related to groups (they are deprecated in proto2) - we are not going to implement full support for features that are already deprecated.

Hi,

Is there a way to generate protostuff pojo from a proto file? The protostuff pojo should implement ProtostuffSerializable.

Our dilemma is we have a proto file and we generate the protostuff POJO counter via this maven plugin:

<plugin>
   <groupId>io.protostuff</groupId>
   <artifactId>protostuff-maven-plugin</artifactId>
   <version>2.2.27</version>
   <executions>
     <execution>
        <id>generate-sources</id>
        <phase>generate-sources</phase>
        <goals>
           <goal>java</goal>
      </goals>
  </execution>
 </executions>
</plugin>

But the POJO generated by this plugin does not implement ProtostuffSerializable. We need it so we can integrate our protostuff pojo to our AKKA system. We currently using this plugin: https://github.com/Inveasy/akka-serialization-protostuff as serialization plugin.

Design extensible template system that enables code generation for any language.

I have this message:

message SomeMessage {
    message SomeInnerMessage {
        optional string someField = 1 [default = "some_value"];
        optional string someOtherField = 2 [default = "some_other_value"];
        required string parameter1 = 3;
        required string parameter2 = 4;
    }
    ...
}

Compiler generates this code:

...
    public static final class SomeInnerMessage
            implements io.protostuff.Message<Request> {

        private static final Request DEFAULT_INSTANCE = newBuilder().build();

        private String someField;

        private String someOtherField;

        private String parameter1;

        private String parameter2;

        private boolean __merge_lock = false;
        private int __bitField0;

        private SomeInnerMessage() {
            this.someField = "";
            this.someOtherField = "";
            this.parameter1 = "";
            this.parameter2 = "";
        }
        ...
     }
...

There are neither default initializers for fields someField, someOtherField for private constructor nor for SomeInnerMessage.Builder private constructor. How to generate classes with default fields values?

Unable to make protected final java.lang.Class java.lang.ClassLoader.findLoadedClass(java.lang.String) accessible: module java.base does not "opens java.lang" to unnamed module @715d6168

java version: 1.8
protostuff-maven-plugin: 2.2.18
plugin configuration:

                io.protostuff
                protostuff-maven-plugin
                2.2.18
                
                    
                        generate-proto-documentation
                        generate-sources
                        
                            html
                        
                        
                            src/main/resources/proto
                            
                                
                                    Index
                                    ${project.basedir}/index.md
                                
                                
                                    Changelog
                                    ${project.basedir}/changelog.md
                                
                            
                        
                    
                
            

I have two files: proto_x.proto and imported.proto, proto_x.proto has import imported.proto.

When I use compiler CLI with the following options: proto_x.proto -g=html -I=src/input -o=src/output, there is no JSON schema of imported.proto was genererated into src/output/data/type.

Did I configure something wrong? Thank you for checking out.

Maps are already supported by parser, this task is about adding support for map to HTML and Java generators.

We need command-line interface with protoc-like syntax.

https://developers.google.com/protocol-buffers/docs/reference/proto3-spec#reserved
https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-3

Added a "reserved" keyword in both proto2 and proto3 syntax. User can use
this keyword to declare reserved field numbers and names to prevent them
from being reused by other fields in the same message.

To reserve field numbers, add a reserved declaration in your message:

message TestMessage {
  reserved 2, 15, 9 to 11, 3;
}

This reserves field numbers 2, 3, 9, 10, 11 and 15. If a user uses any of
these as field numbers, the protocol buffer compiler will report an error.

Field names can also be reserved:

message TestMessage {
  reserved "foo", "bar";
}

reserved = "reserved" ( ranges | fieldNames ) ";"
fieldNames = fieldName { "," fieldName }

Examples:

reserved 2, 15, 9 to 11;
reserved "foo", "bar";

From version 3.5.0 protobuf supports the reserved inside the enum.

Hey,

I just wanted share how to run the compiler from Gradle if you wanted to add it to the wiki for others who might come looking.

Essentially in your Gradle build file you can specify a configuration like so

configurations {
    protoCompile
}

Then add the CLI dependency using that configuration.

dependencies {
    protoCompile "io.protostuff:protostuff-cli:0-SNAPSHOT"
}

Then to run it, just create task and do the following.

task generateSource() {
    ant.java(classname: 'io.protostuff.compiler.cli.ProtostuffCompilerCli', fork: 'true', failOnError: 'true') {
        arg(value: 'MyProto.proto')
        arg(value: '-g=java')
        arg(value: '-I=' + project.projectDir.path + "/../proto/")
        arg(value: '-o=' + project.buildDir.path + "/generated/")
        classpath {
            pathElement(path: configurations.protoCompile.asPath)
        }
    }
}

Then on call the task, I have it as part of the build so on "clean" the generated files are deleted and on build they are re-generated.

/gradlew generateSource

Sample:

message X {
    optional bool y = 1 [default = false];
}

X#toString prints y as "*".

Problem in org.stringtemplate.v4.misc.MapModelAdaptor used by code generator.

public class MapModelAdaptor implements ModelAdaptor {
	@Override
	public Object getProperty(Interpreter interp, ST self, Object o, Object property, String propertyName)
		throws STNoSuchPropertyException
	{
		Object value;
		Map<?, ?> map = (Map<?, ?>)o;
		if ( property==null ) value = map.get(STGroup.DEFAULT_KEY);
		else if ( property.equals("keys") ) value = map.keySet();
		else if ( property.equals("values") ) value = map.values();
		else if ( map.containsKey(property) ) value = map.get(property);
		else if ( map.containsKey(propertyName) ) { // if can't find the key, try toString version
			value = map.get(propertyName);
		}
		else value = map.get(STGroup.DEFAULT_KEY); // not found, use default
		if ( value == STGroup.DICT_KEY ) {
			value = property;
		}
		return value;
	}
}

When it does property lookup in a map, it also checks "default" (STGroup.DEFAULT_KEY) key. Unfortunately it is also used in protobuf as an option key.

As a result, all fields with "default" value will be masked as "*".

When I use protostuff-parser to parse proto file, A io.protostuff.compiler.parser.ParserException raised.
I read the code and found the reason:

The io.protostuff.compiler.model.Proto.getNamespace() implements as this:

@Override
public String getNamespace() {
    if (pkg == null) {
        return ".";
    }
    return "." + pkg.getValue() + ".";
}

And the default value of pkg is protected Package pkg = Package.DEFAULT;, meanwhile the implements of getValue in Package.DEFAULT return ""(empty string).

So when use ImportImpl to parse a proto file without a "package" declared, the Message and other elements in the proto file will be registered as a fullyQualifiedName starts with double point(".."):

// io.protostuff.compiler.parser.TypeRegistratorPostProcessor
private void registerUserTypes(ProtoContext context) {
    final Proto proto = context.getProto();
    List<Message> messages = new ArrayList<>();
    messages.addAll(proto.getMessages());
    // ......
    for (Message type : messages) {
        type.setProto(proto);
        String fullyQualifiedName = proto.getNamespace() + type.getName();
        type.setFullyQualifiedName(fullyQualifiedName);
        context.register(fullyQualifiedName, type);
    }
    // ......
}

As that, When proto file "A" which declares "package" as "aaaa.bbb", imports another proto file "B" which no "package" declared, it will not be able to found Message XXX declared in B, because the XXX is registered as "..XXX", and the look up for XXX is only for ".aaa.bbb.XXX", ".aaa.XXX", ".XXX":

// io.protostuff.compiler.parser.TypeResolverPostProcessor

public static Deque<String> createScopeLookupList(UserTypeContainer container) {
    String namespace = container.getNamespace();
    Deque<String> scopeLookupList = new ArrayDeque<>();
    int end = 0;
    while (end >= 0) {
        end = namespace.indexOf('.', end);
        if (end >= 0) {
            end++;
            String scope = namespace.substring(0, end);
            scopeLookupList.addFirst(scope);
        }
    }
    return scopeLookupList;
}

private UserType resolveUserType(Element source, ProtoContext context, Deque<String> scopeLookupList, String typeName) {
    UserType fieldType = null;
    // A leading '.' (for example, .foo.bar.Baz) means to start from the outermost scope
    if (typeName.startsWith(".")) {
        UserType type = (UserType) context.resolve(typeName);
        if (type != null) {
            fieldType = type;
        }
    } else {
        for (String scope : scopeLookupList) {
            String fullTypeName = scope + typeName;
            UserType type = (UserType) context.resolve(fullTypeName);
            if (type != null) {
                fieldType = type;
                break;
            }
        }
    }
    if (fieldType == null) {
        String format = "Unresolved reference: '%s'";
        throw new ParserException(source, format, typeName);
    }
    return fieldType;
}

I think it is legal to create a proto file without "package". So I wonder:

• If there are a bug with proto file no "package" declared?
• Is it more rational to make the pkg of Proto as null or change the implement of getNamespace to return "." when pkg == Package.DEFAULT?

message.proto

  enum EncodingType {
    kEncodingNone = 0;
    kEncodingLz4 = 1;
  }

java object (generated by compiler)

I am shepherding a change in Protobufs to allow multi-line string values:

protocolbuffers/protobuf#1297

This ticket serves as a marker for the corresponding effort required in this project. I have forked this project and gotten a working version on my fork. I'll submit a PR once things start solidifying in the protobuf project.

In order to follow strict model-view separation, we need a mechanism for extending objects with new properties.

For example, Proto class can be extended with property java_package_path that returns relative package path ("io/protostuff/test/"), formed from java_package option ("io.protostuff.test").

API should be like this:

extendClass(Class<?> classToExtend, String newPropertyName, Function<T, U> f);

Support for parsing/generating code in projects running with JDK 16.

Generate HTML documentation from proto files.

Given enums E1 and E2:

enum E {
  A = 0;
  B = 1;
}

enum E2 {
  A = 0;
  B = 1;
  C = 2;
}

Steps:

1. Serialize a field of type E2 and value = C.
2. Deserialize result into a field of type E1.

Actual result:
Field is not set, getter returns A.

Expected result:
Field is not set, getter returns synthetic value UNRECOGNIZED.

This issue is fixed in proto3, but it makes sense to have this fixed for proto2 too.

Parser should check enum key type and return readable error.

Something like in protoc:

foo.proto:8:3: Key in map fields cannot be enum types.

TODO: revert a4957fb