Giter VIP home page Giter VIP logo

dsl-json's Introduction

DSL-JSON library

Fastest JVM (Java/Android/Scala/Kotlin) JSON library with advanced compile-time databinding support.

Java JSON library designed for performance. Originally built for invasive software composition with DSL Platform compiler.

JVM serializers benchmark results

Distinguishing features

  • works on existing POJO classes via annotation processor
  • performance - faster than any other Java JSON library. On par with fastest binary JVM codecs
  • works on byte level - deserialization can work on byte[] or InputStream. It doesn't need intermediate char representation
  • extensibility - support for custom types, custom analyzers, annotation processor extensions...
  • streaming support - large JSON lists support streaming with minimal memory usage
  • allocation friendly - converters avoid producing garbage
  • minimal size - runtime dependency weights around 450KB
  • no unsafe code - library doesn't rely on Java UNSAFE/internal methods
  • POJO <-> object and/or array format - array format avoids serializing names, while object format can be used in minimal serialization mode
  • legacy name mapping - multiple versions of JSON property names can be mapped into a single POJO using alternativeNames annotation
  • binding to an existing instance - during deserialization an existing instance can be provided to reduce GC
  • generics, builder pattern, factory pattern and ctor with arguments - all relevant initialization methods are supported
  • compile time detection of unsafe conversion - can throw compile time error for conversion which can fail at runtime
  • customizable runtime overheads - works in reflection mode or annotation processor mode. Annotation based POJOs are prepared at compile time
  • support for other library annotations - Jackson and JsonB annotations will be used and compile time analysis can be extended in various ways
  • Scala types support - Scala collections, primitives and boxed primitives work without any extra annotations or configuration
  • Kotlin support - annotation processor can be used from Kotlin. NonNull annotation is supported
  • JsonB support - high level support for JsonB String and Stream API. Only minimal support for configuration
  • compatible with DSL Platform

Upgrade from v1

v1 core library was targeting Java6, while v2 targets Java8. So there is no need anymore for dsl-json-java8 project and instead just dsl-json project should be referenced. From v2 only a single annotation processor is supported, which was previously in java8 project.

Joda and some other specific libraries were removed. If you still need converters for those types, you need to include them in your project.

Few methods which were previously marked as @Deprecated are now removed, mostly around creating JsonWriter and JsonReader which should be created through DslJson instead of instatianted directly.

@CompiledJson annotation

Annotation processor works by analyzing Java classes and its explicit or implicit references. Processor outputs encoding/decoding code/descriptions at compile time. This avoids the need for reflection, provides compile time safety and allows for some advanced configurations.

By default, library only searches for @CompiledJson annotation, but it can be configured to search for @JacksonCreator and @JsonbCreator.
Converters will be created even for dependent objects which don't have relevant annotation. This can be used to create serializers for pre-existing classes without annotating them.

There are 2 main ways how generated code/manual services are detected:

  • with lookups from META-INF/services through ServiceLoader during DslJson initialization
  • by probing for name conventions: package._NAME_DslJsonConverter when required

Annotation processor

Annotation processor provides most features and flexibility, due to integration with runtime analysis and combining of various generic analysis. Bean properties, public fields, classes without empty constructor, factories and builder patterns are supported. Package private classes and factory methods can be used. Array format can be used for efficient payload transfer.

To use annotation processor it is sufficient to just reference the library:

<dependency>
  <groupId>com.dslplatform</groupId>
  <artifactId>dsl-json</artifactId>
  <version>2.0.2</version>
</dependency>

For use in Android, Gradle can be configured with:

android {
  compileOptions {
    sourceCompatibility 1.8
    targetCompatibility 1.8
  }
}
dependencies {
  compile 'com.dslplatform:dsl-json:2.0.2'
  annotationProcessor 'com.dslplatform:dsl-json:2.0.2'
  provided 'javax.json.bind:javax.json.bind-api:1.0'
}

Project examples can be found in examples folder

Custom types in compile-time databinding

Types without built-in mapping can be supported in three ways:

  • by defining custom conversion class and annotating it with @JsonConverter
  • by defining custom conversion class and referencing it from property with converter through @JsonAttribute
  • by implementing JsonObject and appropriate JSON_READER

Custom converter for java.time.LocalTime can be found in example project Annotation processor will check if custom type implementations have appropriate signatures.

@JsonConverter which implements Configuration will also be registered in META-INF/services which makes it convenient to setup initialization.

All of the above custom type examples work out-of-the-box.

Custom converter is a class with 2 static methods, eg:

public static abstract class LocalTimeConverter {
	public static LocalTime read(JsonReader reader) throws IOException {
		return LocalTime.parse(reader.readSimpleString());
	}
	public static void write(JsonWriter writer, LocalTime value) {
		writer.writeString(value.toString());
	}
}

For mutable objects 3rd optional method is supported bind

Optional default value can be specified in converters which will be used for missing properties and during serialization for omit values check

public static abstract class PiConverter {
	//... read and write
	public static final double JSON_DEFAULT = 3.14159;
}

This feature is useful for introducing new properties into classes which might not be yet sent, as a way to specify expected value for such new fields.

@JsonAttribute features

DSL-JSON property annotation supports several customizations/features:

  • name - define custom serialization name
  • alternativeNames - different incoming JSON attributes can be mapped into appropriate property. This can be used for simple features such as casing or for complex features such as model evolution
  • ignore - don't serialize specific property into JSON
  • nullable - tell compiler that this property can't be null. Compiler can remove some checks in that case for minuscule performance boost
  • mandatory - mandatory properties must exist in JSON. Even in omit-defaults mode. If property is not found, IOException will be thrown
  • index - defines index order used during serialization or can be used for array format
  • hashMatch - DSL-JSON matches properties by hash values. If this option is turned off exact comparison will be performed which will add minor deserialization overhead, but invalid properties with same hash names will not be deserialized into "wrong" property. In case when model contains multiple properties with same hash values, compiler will inject exact comparison by default, regardless of this option value.
  • converter - custom conversion per property. Can be used for formatting or any other custom handling of JSON processing for specific property
  • typeSignature - disable inclusion of $type during abstract type serialization. By default, abstract type will include additional information which is required for correct deserialization. Abstract types can be deserialized into a concreted type by defining deserializeAs on @CompiledJson which allows the removal of $type during both serialization and deserialization

@JsonValue enum feature

Library supports converting enum as custom value. To use such feature @JsonValue annotation must be placed on method or field.

JSON pretty print

Formatted output with alignments and newlines can be created via PrettifyOutputStream.

dslJson.serialize(instance, new PrettifyOutputStream(outputStream));

External annotations

For existing classes which can't be modified with @JsonAttribute alternative external annotations are supported:

Nullability annotations

During translation from Java objects into DSL schema, existing type system nullability rules are followed. With the help of non-null annotations, hints can be introduced to work around some Java nullability type system limitations. List of supported non-null annotations can be found in processor source code

Nullable annotations can be disabled via configuration parameter dsljson.nullable=false. This might be useful if you want to handle null checks after deserialization.

Property aliases

Annotation processor supports external annotations for customizing property name in JSON:

  • com.fasterxml.jackson.annotation.JsonProperty
  • com.google.gson.annotations.SerializedName

Those annotations will be translated into specialized DSL for specifying serialization name.

Ignored properties

Existing bean properties and fields can be ignored using one of the supported annotations:

  • com.fasterxml.jackson.annotation.JsonIgnore
  • org.codehaus.jackson.annotate.JsonIgnore

Required/mandatory properties

Jackson required = true can be used to fail if property is missing in JSON. DSL-JSON has distinct behavior for required (non-null) and mandatory specification.

Mandatory means that property must be present in JSON. If it's not present an error will be thrown during deserialization. Non-null means that property cannot be null. But most types have default values and if omitted will assume this default value, eg:

  • String - empty string
  • Number - 0
  • Boolean - false
  • etc...

Some types don't have default value and for them, non-null also implies mandatory. This behavior can be controlled via several options.

Serialization modes

Library has several serialization modes:

  • minimal serialization - omits default properties which can be reconstructed from schema definition
  • all properties serialization - will serialize all properties from schema definition
  • array format - object will be serialized as array without property names

Best serialization performance can be obtained with combination of minimal serialization and minified property names/aliases or array format. Object and array formats can be combined. POJO can support both formats at once.

Benchmarks

Independent benchmarks can validate the performance of DSL-JSON library:

Reference benchmark (built by library authors):

Runtime analysis

Library has built-in runtime analysis support, so library can be used even without compile time databinding or it can just add additional runtime support alongside compile-time databinding (default behavior). Runtime analysis is required for some features such as generics which are not known at compile time. Runtime analysis works by lazy type resolution from registered converters, eg:

private final DslJson.Settings settings = runtime.Settings.withRuntime().includeServiceLoader();
private final DslJson<Object> json = new DslJson<Object>(settings);

Best practices

Reusing reader/writer.

JsonWriter has two modes of operations:

  • populating the entire output into byte[]
  • targeting output stream and flushing local byte[] to target output stream

JsonWriter can be reused via reset methods which binds it to specified target. When used directly it should be always created via newWriter method on DslJson instance. Several DslJson serialize methods will reuse the writer via thread local variable. When using JsonWriter via the first mode, result can be copied to stream via .toStream(OutputStream) method.

DslJson<Object> json = ... // always reuse
OutputStream stream = ... // stream with JSON in UTF-8
json.serialize(pojo, stream); //will use thread local writer

JsonReader can process byte[] or InputStream inputs. It can be reused via the process methods. When calling DslJson deserialize methods often exists in two flavors:

  • with byte[] argument, in which case a new JsonReader will be created, but for best performance byte[] should be reused
  • without byte[] argument in which case thread local reader will be reused

For small messages it's better to use byte[] API. When reader is used directly it should be always created via newReader method on DslJson instance.

DslJson<Object> json = ... // always reuse
InputStream stream = ... // stream with JSON in UTF-8
POJO instance = json.deserialize(POJO.class, stream); //will use thread local reader

Binding

JsonReader has iterateOver method for exposing input collection as consumable iterator.

DslJson<Object> json = new DslJson<Object>(); //always reuse
byte[] bytes = "{\"number\":123}".getBytes("UTF-8");
JsonReader<Object> reader = json.newReader().process(bytes, bytes.length);
POJO instance = new POJO(); //can be reused
POJO bound = reader.next(POJO.class, instance); //bound is the same as instance above

Binding API is available which can reuse instances for deserialization. This is supported in converters via 3rd optional JSON_BINDER member or bind method, eg:

class POJO {
  public String mutableValue;
}
class PojoConverter {
  public POJO bind(JsonReader reader, POJO instance) {
    if (instance == null) instance = new POJO();
    instance.mutableValue = StringConverter.read(reader);
  }
}

Limits

Library has various configurable limits built-in to protect against malicious input:

  • default of 512 digits
  • default of 128MB strings

String API

DSL-JSON works on byte level. To discourage use of String for JSON processing there is no high level String API. If one really needs to use String, manual conversion is required.

DslJson<Object> json = new DslJson<Object>();
byte[] bytes = "{\"number\":123}".getBytes("UTF-8"); //convert string to UTF-8 bytes
POJO instance = json.deserialize(POJO.class, bytes, bytes.length);
ByteArrayOutputStream os = new ByteArrayOutputStream();
json.serialize(instance, os);
String outputAsString = os.toString("UTF-8"); //convert stream to string

Scala support

Scala types can be used. They will be analyzed at runtime with. Scala specific behaviour:

  • Option[_] - means that JSON attribute can be null. If type is not an Option and null is found, IOException will be thrown
  • Container[Primitive] - eg: Option[Int] - will behave as Option<int> and thus it will avoid wrong type decoding issues
  • name: Type = Default - will be used to imply if attribute can be omitted from JSON - in which case the specified default value will be used (default values are static at analysis time)
  • tuples - will be encoded/decoded in Array format (without property names)

To avoid some Java/Scala conversion issues it's best to use Scala specific API via

import com.dslplatform.json._ // import pimping
val dslJson = new DslJson[Any]()
//use encode pimp to correctly analyze types (this will mostly provide some performance benefits)
dslJson.encode(instance, ...)
//use decode pimp to correctly analyze types (this will avoid some issues with nested classes and missing metadata)
val result = dslJson.decode[TargetType](...) 

For SBT dependency can be added as:

libraryDependencies += "com.dslplatform" %% "dsl-json-scala" % "2.0.2"

Kotlin support

Kotlin has excellent Java interoperability, so annotation processor can be used as-is. When used with Gradle, configuration can be done via:

plugins {
  kotlin("kapt") version "1.8.0"
}
dependencies {
  implementation("com.dslplatform:dsl-json:2.0.2")
  kapt("com.dslplatform:dsl-json:2.0.2")
}

FAQ

Q: What is TContext in DslJson and what should I use for it?
A: Generic TContext is used for library specialization. Use DslJson<Object> when you don't need it and just provide null for it.

Q: How do I use DSL-JSON with String?
A: Don't do that. You should prefer Streams or byte[] instead. If you really, really need it, it is mentioned in this README.

Q: Why is DSL-JSON faster than others?
A: Almost zero allocations. Works on byte level. Better algorithms for conversion from byte[] -> type and vice-versa. Minimized unexpected branching. Reflection version is "comparable" with Jackson performance. Extra difference comes from compile-time databinding.

Q: I get compilation error when annotation processor runs. What can I do?
A: Common error is missing dependency on Java 9+ for annotation marker. You can add such dependency on configure compiler arguments to exclude it via dsljson.generatedmarker. Otherwise, it's best to inspect the generated code, look if there is some configuration error, like referencing class without sufficient visibility. If there is nothing wrong with the setup, there might be a bug with the DSL-JSON annotation processor in which case it would be helpful to provide a minimal reproducible

Q: Can you please help me out with...?
A: There is only so many hours in a day. You can support the library by asking for support contract with your company.

dsl-json's People

Contributors

abnud1 avatar abnud11 avatar benco-amzn avatar felixbarny avatar hkratz avatar jdpgrailsdev avatar johannes-b2c2 avatar josealavez avatar jpe42 avatar kkoehler avatar lyrachord avatar melezov avatar plokhotnyuk avatar qny31541 avatar qudratillo avatar schernic avatar umutsahin avatar varahash avatar zapov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dsl-json's Issues

Deal with Umlaut (รค, รถ, รผ, or รŸ) in input

The following JSON breaks the parser with
SEVERE: IOException parsing entry: Unexpected end of JSON input

{
	"@timestamp": "2016-09-05T12:44:46.000Z",
	"url": {
		"analyzed": "/-/-/call-1111",
		"path": "/-/-/call-1111",
		"query": {
			"was": "sรคgewerk gmbh"
		}
	},
	"event": "view",
	"origin": "detail",
	"session": ["-9035111172716044880", "6141211111163737100", "-6891111112685400843", "-275111111849250600"],
	"@version": "1"
}

(The was field contains user input which includes HTML-valid Umlaute as รค,รผ,รถ, or รŸ.)

Instance binding

Add support for binding instance to input. This can reduce GC by reusing instances during low level processing

Improve usage of generics to eliminate "unchecked" warnings and other errors

  1. There are few places where usage of raw generic type leads to unchecked warnings, e.g:
public MyConfiguration implements Configuration {
	void configure(DslJson json) {
               json.registerReader(...)
        }
}

since DslJson is generic, calling methods on json object produces warning. As solution we should replace raw generic types by unbounded wildcard parmeterized type (e.g DslJson<?>) or where possible by concrete parmeterized type (e.g DslJson<T>).

  1. Replace usage of generic arrays with generic list. Since java does not allow create generic arrays, we have to cast object array to generic array which always produce unchecked warnings. eg
public static <D> ObjectFormatDescription<D, D> create(
			final Class<D> manifest,
			final InstanceFactory<D> newInstance,
			final JsonWriter.WriteObject[] encoders,
			final DecodePropertyInfo<JsonReader.BindObject>[] decoders,
			final DslJson json,
			final boolean skipOnUnknown) { ... }

ObjectFormatDescription<MyClass, MyClass> description = ObjectFormatDescription.create(
				MyClass.class,
				MyClass::new,
				new JsonWriter.WriteObject[0],
				new DecodePropertyInfo[] {
					Settings.<MyClass, Integer>createDecoder(...),
					Settings.<MyClass, String>createDecoder(...)
				},
				dslJson, false);

here we should use final List<DecodePropertyInfo<JsonReader.BindObject<D>>> decoders and instead of new DecodePropertyInfo[] {..} use Arrays.asList(...), this is clean, type safe and warning free way.

  1. Use upper bound wildcard in some cases, e.g
public Settings<TContext> resolveReader(ConverterFactory<JsonReader.ReadObject> reader) { ... }

class MyReader implements JsonReader.ReadObject { ... }
class MyConverterFactory implements ConverterFactory<MyReader> { ... }

json.resolveReader(new MyConverterFactory()); // error

to resolve the error above, we should change signature to resolveReader(ConverterFactory<? extends JsonReader.ReadObject> reader)

This is not complete list, perhaps there are other similar issues.

Downloaded compiler executable is not kept

Currently the dsl-platform.com website is down, meaning I cannot download the compiler tool as required. It would be good if the executable is cached in case of server downtime.

Improve error reporting

Instead of just specifying byte where the error occurred, try to show line number/column number also when possible.
It's possible to do so when input is byte array or the stream is still on the first buffer (which should be most of the time)

Several long strings with many ASCII letters following a special character cannot be parsed

JsonReader.parseString() contains a bug related to the variable _tmpLen, which is at first used to keep track of how many bytes are still ahead in the buffer, and later to reflect the size of the temporary buffer. But it isnt't set to the size of the temporary buffer when it changes its role, so whenever a backslash or UTF-8 multibyte character was encountered, the temporary buffer will often unnecessarily double its size, which will soon lead to OutOfMemoryError.

	if (i == _tmp.length) {
		_tmp = chars = Arrays.copyOf(chars, chars.length * 2);
	}
	_tmpLen = _tmp.length; // This line needs to be inserted!
	currentIndex = ci;
	int soFar = --currentIndex - startIndex;

Cannot serialize nested collections

I just ran into an issue while handling some map data from another library - nested lists can't be serialized.

My test code:

DslJson dsl = new DslJson();
dsl.serializeMap(
        Collections.singletonMap("x",
                Collections.singletonList(Collections.singletonList("Hello"))
        ),
        dsl.newWriter()
);

I think the issue is around here

Disabling including of type information during serialization

I'm currently using dsl-json in an API server. We have several types of response messages with different parameters, all of which implement the Response interface. I believe type information is included automatically because we're using an abstract type, but I'm looking for some way to disable that.

@CompiledJson
public class ApiOutput {
    public Response data;
    public Integer status_code;
    ....
}

@CompiledJson
public class ResultResponse implements Response {
    public String result = "success";
}

The output JSON of serialization:

{
    "data": {
        "$type": "responses.ResultResponse",
        "result": "success"
    },
    "status_code": 200,
    ...
}

Is there some way to disable the $type field during serialization? Maybe some annotation parameter?

CompiledJsonProcessor error

Hi๏ผŒi got an error while compiling code,but i don't know where it come from.

Warning:Supported source version 'RELEASE_6' from annotation processor 'com.dslplatform.json.CompiledJsonProcessor' less than -source '1.8'

Feature Request: Register Interfaces on ExternalSerialization#setup

For @CompiledJson classes, would it be possible to register generated JsonReader/Writers with the interface they implement as well? The behavior I am observing is that only the annotated class is registered (DslJson#registerReader). Maybe adding a list of superclasses to the @CompiledJson annotation would be the easiest route?

Currently, I have an if statement for each expected interface and then manually pass the corresponding implementation class.

Support for non-serialization of null fields

Would it be possible to add a setting to avoid the serialization of a field if it is null? Or to conditionally apply the @ignore annotation.

Currently I am serializing a POJO that has nullable fields with a requirement to only serialize non-null values. In Jackson it could be achieved with a @JsonInclude(JsonInclude.Include.NON_NULL) annotation.

JsonStreamReader.getLastName returns incorrect data sometimes

From what I can gather, it seems like this happens after JsonStreamReader reads from the stream into the buffer - in-between the name and value of the json property. The buffer doesn't contain the name anymore, so it can't be extracted properly (specifically here).

Annotations with Android

I figured that the current version of Gradle/Android uses jack and is incompatible with the android-apt plugin. How do i use dsl-json in current android. The provided sample is also "outdated"

Support runtime analysis

Most of the performance comes from fast converters. Reflection based databinding should be supported to avoid the need for custom forks due to .net dependency in databinding

External annotation support

In the readme it says that this library supports external annotations like Jackson's @JsonProperty. Can't find any example of an annotation being used to change the name used for deserialization/serialization declared outside of the model class itself.

In Jackson I'd just create a new mixin class and register it with my object mapper. Is their similar support for dsl-json?

Thanks

How does dsl-json deal with circular dependency?

I'm evaluating dsl-json and considering migrating from jackson. The performance I saw on a few benchmarks seemed very impressive and I'd love to play with it a bit more. However, one of my core needs is to deal with graph ser/deser.

Does the lib provide ways to handle circular dependency out-of-the-box? I couldn't find anything on docs or examples. If so, how does it work? If not, is there any way to solve that?

Deserializing in real time, mutable objects

I'm deserializing an array of objects as follows:

// This is cached, only one instance created
DslJson<Object> dslJson = new DslJson<>(Settings.withRuntime().includeServiceLoader());
JsonReader reader = dslJson.newReader();

// I run this part in a loop thousands of times per second
HttpResponse<InputStream> httpResponse = request.asBinary();
reader.process(httpResponse.getBody());
if (reader.getNextToken() == '[') {
    do {
        // Read beginning of object
        reader.getNextToken(); // {

        // Read id
        reader.getNextToken(); // "
        reader.fillName();     // id
        reader.getNextToken(); // "
        String id = reader.readString(); // foo
        reader.getNextToken(); // ,

        // Read price
        reader.getNextToken(); // "
        reader.fillName();     // price
        reader.getNextToken(); // "
        double price = NumberConverter.deserializeDouble(reader);

        // Read ending of object
        reader.getNextToken(); // }

        // Update values
        update(id, price);
    } while (reader.getNextToken() == ',');
}

It's working fine, but I'm not sure this is the fastest way of doing this as I barely know DSL-JSON api.

How can I increase the performance?

Deserializing abstract class based on another field value?

I have a field that depends on another field's value to define what type it is, something like this:

@CompiledJson
public class MyMessage {
    public String valueOne;
    public String valueTwo;
    
    public Thing thing;
    public ThingType type;
}

public enum ThingType { Abc, Xyz }

public interface Thing {}
public class AbcThing implements Thing {}
public class XyzThing implements Thing {}

I'm just wondering how the best way to go about this is. I'm not really certain of how I can use JsonConverter for this - because the field is dependent on another field being read first.

apt fails at processing fields starting with _

100% repro with the 2 simple following files:

build.gradle

plugins {
    id "net.ltgt.apt" version "0.6"
}

apply plugin: 'java'

sourceCompatibility = 1.8
targetCompatibility = 1.8

repositories {
    mavenCentral()
}

sourceSets {
  main {
    java {
      srcDir 'src'
    }
  }
}

dependencies {
    // DSL-json
    compile group: 'com.dslplatform', name: 'dsl-json', version: '1.1.1'
    apt group: 'com.dslplatform', name: 'dsl-json-processor', version: '1.2'

    // Test
    testCompile group: 'junit', name: 'junit', version: '4.12'
}


test {
    testLogging {
        exceptionFormat = 'full'
        // showStandardStreams = true
    }
}

in src/thepack/TheObj.java

package thepack;

import com.dslplatform.json.CompiledJson;

@CompiledJson
public class TheObj {

  public String _id;

}

then: gradle build

:compileJava
~/dsljsonbug/build/generated/source/apt/main/dsl_json/json/ExternalSerialization.java:44: error: cannot find symbol
            if (self.id != null) {
                    ^
  symbol:   variable id
  location: variable self of type TheObj
~/dsljsonbug/build/generated/source/apt/main/dsl_json/json/ExternalSerialization.java:47: error: cannot find symbol
                sw.writeString(self.id);
                                   ^
  symbol:   variable id
  location: variable self of type TheObj
~/dsljsonbug/build/generated/source/apt/main/dsl_json/json/ExternalSerialization.java:55: error: cannot find symbol
            if (self.id != null) {
                    ^
  symbol:   variable id
  location: variable self of type TheObj
~/dsljsonbug/build/generated/source/apt/main/dsl_json/json/ExternalSerialization.java:57: error: cannot find symbol
                sw.writeString(self.id);
                                   ^
  symbol:   variable id
  location: variable self of type TheObj
~/dsljsonbug/build/generated/source/apt/main/dsl_json/json/ExternalSerialization.java:138: error: cannot find symbol
        instance.id = _id_;
                ^
  symbol:   variable id
  location: variable instance of type TheObj
5 errors
:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 6.618 secs

Versions:

> java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

> mono --version
Mono JIT compiler version 4.2.3 (Stable 4.2.3.4/832de4b Wed Mar 30 13:57:48 PDT 2016)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
  TLS:           normal
  SIGSEGV:       altstack
  Notification:  kqueue
  Architecture:  amd64
  Disabled:      none
  Misc:          softdebug
  LLVM:          supported, not enabled.
  GC:            sgen

Support builder classes for immutable objects

I would like to add dsl-json support to https://github.com/soabase/soabase-halva/blob/master/halva/src/main/java/io/soabase/halva/caseclass/README.md . halva generates immutable classes that are paired with builder classes. Jackson support for the builder pattern can be implemented with @JsonDeserialize.

Example: https://github.com/soabase/soabase-halva/blob/master/examples/example-generated/JsonTestCase.java

Is there an equivalent way to specify a custom builder object with dsl-json? Default compilation results in:

CompiledJson requires accessible public no argument constructor, therefore 'com.dslplatform.maven.Example.ExampleCase' requires public no argument constructor

TypeSignature.EXCLUDE not working on getter/setter methods

I just ran into this issue. It seems that the type signature is included even if EXCLUDE is used for getter/setter methods.

TestObject:

@CompiledJson
public class TestObject {
    private Thing value;

    public TestObject() {
    }

    @JsonAttribute(typeSignature = CompiledJson.TypeSignature.EXCLUDE)
    public Thing getValue() {
        return value;
    }

    @JsonAttribute(typeSignature = CompiledJson.TypeSignature.EXCLUDE)
    public void setValue(Thing value) {
        this.value = value;
    }
}

Main:

public class Main {
    public static void main(String arg[]) {
        TestObject x = new TestObject();
        x.setValue(new BlueThing());
        System.out.println(serialize(x));
    }
}

Output:
{"value":{"$type":"test_env.thing.BlueThing","abc":"aaa"}}

JsonAttribute ignored on interface method implementations

There seems to be a new issue with JsonAttribute in the latest version.

public interface TestInt {
    long getValue();
    void setValue(long v);
}
@CompiledJson
public class TestObject implements TestInt {
    private long value;

    @Override
    @JsonAttribute(name = "my_data")
    public long getValue() {
        return value;
    }

    @Override
    @JsonAttribute(name = "my_data")
    public void setValue(long v) {
        this.value = v;
    }
}

Output is: {"value":123}, but should be {"my_data":123}

Add support for runtime analysis at Java 6

This will allow use reflection data binding on platform which currently does not support Java8 (e.g. Android prior API 24).

I propose to move appropriate classes from 'dsl-json-java8' module to core module 'dsl-json' or we can create new one 'dsl-json-reflection'.

It seems relatively easy to do. We just need to replace usage of 'java.lang.function.*' with our custom interfaces. And keep ImmutableAnalyzer class in 'dsl-json-java8' module, since parameter names are not supported prior Java8.

error: Failed saving compiled json serialization files

I'm getting this error when trying to build an Android (Kotlin) project with dsl-json.

error: Failed saving compiled json serialization files

This is my build.gradle:

dependencies {
  ...
  kapt "com.dslplatform:dsl-json-processor:1.7.3"
  implementation "com.dslplatform:dsl-json-java8:1.7.3"
}

These are the entities I'm trying to deserialize:

@CompiledJson
data class CityJson(
        var id:         Long        = 0,
        var name:       String      = "",
        var country:    String      = "",
        var coord:      CoordJson   = CoordJson(0.0, 0.0)
)

@CompiledJson
data class CoordJson(
        var lat: Double = 0.0,
        var lon: Double = 0.0
)

and this is how I'm trying to do use dsl-json:

val input = ctx.resources.openRawResource(R.raw.city_list)
val dslJson = DslJson<Any>(Settings.withRuntime<Any>().includeServiceLoader())
val reader =  dslJson.newReader().process(input)
var city = reader.next(CityJson::class.java)

while (city != null) {
  ...
  city = reader.next(CityJson::class.java)
}

Link to debug log (with gradle --stacktrace option and dsl-json.loglevel=DEBUG):
https://gist.github.com/dvhp/d1be8a8bebad9f0539b4a50b16374cd8

Fails to parse valid JSON

I'm trying to parse JSON like:

{
  "success": 1,
  "return": {
    "343152": {
      "pair": "abcd",
      "type": "xyz",
      "start_amount": 13.345,
      "amount": 12.345,
      "rate": 485,
      "timestamp_created": 1342448420,
      "status": 0
    }
  }
}

But it fails with:

Exception in thread "main" java.io.IOException: Unexpected end of JSON input
	at com.dslplatform.json.JsonReader.read(JsonReader.java:292)
	at com.dslplatform.json.JsonReader.getNextToken(JsonReader.java:684)
	at com.dslplatform.json.ObjectConverter.deserializeMap(ObjectConverter.java:122)
	at com.dslplatform.json.ObjectConverter$4.read(ObjectConverter.java:31)
	at com.dslplatform.json.ObjectConverter$4.read(ObjectConverter.java:28)
	at com.dslplatform.json.DslJson.deserialize(DslJson.java:1263)
	at com.example.Test.main(Test.java:16)

I've tried to simplify the JSON to find what exactly causes the problem to:

{
  "a": 1,
  "b": {
    "c": {
      "d": "e"
    }
  }
}

but now it fails with:

Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
	at java.lang.String.<init>(String.java:196)
	at com.dslplatform.json.DslJson$SimpleStringCache.createAndPut(DslJson.java:539)
	at com.dslplatform.json.DslJson$SimpleStringCache.get(DslJson.java:530)
	at com.dslplatform.json.JsonReader.readKey(JsonReader.java:929)
	at com.dslplatform.json.ObjectConverter.deserializeMap(ObjectConverter.java:120)
	at com.dslplatform.json.ObjectConverter.deserializeObject(ObjectConverter.java:87)
	at com.dslplatform.json.ObjectConverter.deserializeMap(ObjectConverter.java:125)
	at com.dslplatform.json.ObjectConverter$4.read(ObjectConverter.java:31)
	at com.dslplatform.json.ObjectConverter$4.read(ObjectConverter.java:28)
	at com.dslplatform.json.DslJson.deserialize(DslJson.java:1263)
	at com.example.Test.main(Test.java:16)

My Java code is :

String json = "<ONE OF THE ABOVE SNIPPETS>";
DslJson<Object> dsl = new DslJson<>();
final Map deserialize = dsl.deserialize(Map.class, json.getBytes(StandardCharsets.UTF_8), 16);
System.err.println("des\n" + deserialize);

Mark public api with nullability annotations

We can use Jsr305 as base

Create custom NonNullApi annotation, which makes all parameters and return values non-nullable by default, e.g.

@Target(ElementType.PACKAGE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Nonnull
@TypeQualifierDefault({ElementType.METHOD, ElementType.PARAMETER})
public @interface NonNullApi {
}

then annotate package with it:

// file package-info.java
@NonNullApi
package com.dslplatform.json;

then annotate only nullable parameters or return values

import javax.annotation.Nullable;

public class DslJson<TContext> {
    @Nullable
    public JsonWriter.WriteObject<?> tryFindWriter(final Type manifest) {
    }
}

Mandatory enum fields cause invalid Java syntax to be generated

Using enums with the mandatory attribute seems to cause invalid Java to be generated. I tested with a small sample below:
(dsl-json-processor version 1.4.5)

MyObject.java:

import com.dslplatform.json.CompiledJson;
import com.dslplatform.json.JsonAttribute;

@CompiledJson
public class MyObject {
    @JsonAttribute(mandatory = true)
    public MyEnum value;
}

MyEnum.java:

public enum MyEnum {
    Hello,
    World
}

ExternalSerialization.java error lines:

	static void __serializeJsonObjectMinimal(final MyObject self, com.dslplatform.json.JsonWriter sw, boolean hasWrittenProperty) {
		
		
		else {
			hasWrittenProperty = true;
			sw.writeAscii("\"value\":null", 12);
		}
	}

The inserted "else" here is causing a syntax error during compilation.

String character after emoji when deserialize a json

Hi, I tried to parse the json like the following one by the newest version 1.7.1.

{"1":"457899683877105664","2":2084,"3":{"safe":0.11111111,"blessed":0.11111111,"easter":0.11111111,"hope":0.11111111},"4":"I hope everyone has a safe and blessed Easter :) ๐Ÿ’—"}

My Code is as below:

public class TopicHandler {
    private static final DslJson.Settings settings = Settings.withRuntime().allowArrayFormat(true).includeServiceLoader();
    private static final DslJson<Object> json = new DslJson<Object>(settings);
    .....
    public void handle(RoutingContext routingContext) {
        Map<String, Object> mapContent2 = json.deserialize(Map.class, str.getBytes(), str.getBytes().length);
        System.out.println(mapContent2.get("4").toString());
    }
}

It shows me a strange character after the emoji.

I hope everyone has a safe and blessed Easter :) ๐Ÿ’—๏’—

Do you know why? It seems every emoji will be with a strange character after deserialization

Accept empty arrays (ACCEPT_EMPTY_ARRAY_AS_NULL_OBJECT)

The following JSON breaks the parser:

{
	"@timestamp": "2016-09-05T12:44:46.000Z",
	"url": {
		"analyzed": "/-/-/call-1111",
		"path": "/-/-/call-1111",
		"query": []
	},
	"event": "view",
	"origin": "detail",
	"session": ["-9035111172716044880", "6141211111163737100", "-6891111112685400843", "-275111111849250600"],
	"@version": "1"
}

(Query is a nested JSON object, but when it's empty it is written out as an empty array.)
It would be nice to have an option like ACCEPT_EMPTY_ARRAY_AS_NULL_OBJECT when deserializing JSON.

incorrect error message on field with unsupported type

The error message states:
error: Specified type not supported. If you wish to ignore this property, use one of the supported JsonIgnore annotations (such as Jackson @JsonIgnore or DSL-JSON @JsonAnnotation(ignore = true) on field
but the DSL-JSON annotation type is JsonAttribute, not JsonAnnotation.

Add support for multi module projects

Problem:
Consider multi module project, consists of 'app' and 'lib1' modules, where 'app' module depends on 'lib1'. Each module has POJO classes which annotated with 'CompiledJson'. When we compile 'app' module, the 'dsl_json_Annotation_Processor_External_Serialization' is generated by annotation processor at the top level. But the same file also generated for 'lib1' module. As result we are unable to use generated code from 'lib1' module because, there is class collision.

Solution:
I propose to generate converters for each POJO class, instead of single fat 'dsl_json_Annotation_Processor_External_Serialization' class.
For example if we have 'User' class inside 'com.example.app' package, then generate 'User_dslJsonConverter' class inside same package and register it into 'META-INF/services/com.dslplatform.json.CompiledJson'

WOW,ERROR

DslJson dslJson = new DslJson<>();
JsonWriter writer2 = dslJson.newWriter();
List o = new ArrayList();
o.add("Hello Word!");
o.add(1);
o.add(false);

dslJson.serialize(writer2, o);
System.out.println(writer2.toString());

how could this happen ?

string buffer limit exceeded error

I tried the new version 1.7.2. The problem about json with emoji has gone.

However, after handling some JSON strings, it continued raising the following error even if the JSON is a very short string.

Unable to process input JSON. Maximum string buffer limit exceeded: 134217728

I print the json when it raise error and it is as simple as {"458165560803868672":{"1":7722,"2":"RT @SivuNgcaba: \"Things we buy to cover up what's inside, coz they made us hate ourselves and love their wealth.\""}}

I increased the string buffer limitation to 1GB, the whole system will raise heap size is over even if I set the -Xmx=3000m.

Do you have any clue what happened?

I tried 1.7.1 and it is the same.

Support for deseralization with no type information.

Jackson will default to using LinkedHashMap to represent JSON where there is no type information given, or the type information is given as Object.class.

Is there any support for this type of behaviour with dsl-json?

I am looking to deserialize json where a field may have varying types. For example, in my model class I have an instance variable declared as Map<String, Object> fieldName. The object value may be represented in the JSON as a string, int, object or array etc. Does dsl-json support a method of generically deserializing this into some sort of structure so it can be accessed from the instance variable.

Here is an example of some JSON that I hope to deserialize into a Map<String, Object>

 "fieldName": {
            "alwaysString": {
                "fieldInsideObject": 5
            },
            "anotherString": 40
        }

Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.