Comments (6)
Interesting, did you see any related changes after 0.1.9?
from avro-util.
No, I have no idea what has happened so far. @gaojieliu
from avro-util.
Could you share the generated fast-serializers with 0.1.9 and the latest version?
From the generated classes, it should be straightforward to tell the differences.
from avro-util.
I have made a similar change for Enum Schema (Avro 1.4) in FastSerializerGenerator like FastDeserializerGenerator in #49 .
This is benchmark results. It has become a little better.
Benchmark Mode Cnt Score Error Units
FastAvroSerdesBenchmark.testAvroSerialization avgt 5 7560.036 ± 690.972 ns/op
FastAvroSerdesBenchmark.testAvroSerialization:·gc.alloc.rate avgt 5 406.696 ± 36.323 MB/sec
FastAvroSerdesBenchmark.testAvroSerialization:·gc.alloc.rate.norm avgt 5 3384.000 ± 0.001 B/op
FastAvroSerdesBenchmark.testAvroSerialization:·gc.churn.PS_Eden_Space avgt 5 389.452 ± 865.498 MB/sec
FastAvroSerdesBenchmark.testAvroSerialization:·gc.churn.PS_Eden_Space.norm avgt 5 3209.487 ± 7137.196 B/op
FastAvroSerdesBenchmark.testAvroSerialization:·gc.churn.PS_Survivor_Space avgt 5 0.043 ± 0.313 MB/sec
FastAvroSerdesBenchmark.testAvroSerialization:·gc.churn.PS_Survivor_Space.norm avgt 5 0.358 ± 2.583 B/op
FastAvroSerdesBenchmark.testAvroSerialization:·gc.count avgt 5 4.000 counts
FastAvroSerdesBenchmark.testAvroSerialization:·gc.time avgt 5 14.000 ms
FastAvroSerdesBenchmark.testFastAvroSerialization avgt 10 7464.840 ± 185.288 ns/op
FastAvroSerdesBenchmark.testFastAvroSerialization:·gc.alloc.rate avgt 10 407.911 ± 9.827 MB/sec
FastAvroSerdesBenchmark.testFastAvroSerialization:·gc.alloc.rate.norm avgt 10 3352.000 ± 0.001 B/op
FastAvroSerdesBenchmark.testFastAvroSerialization:·gc.churn.PS_Eden_Space avgt 10 400.763 ± 324.879 MB/sec
FastAvroSerdesBenchmark.testFastAvroSerialization:·gc.churn.PS_Eden_Space.norm avgt 10 3281.003 ± 2660.225 B/op
FastAvroSerdesBenchmark.testFastAvroSerialization:·gc.churn.PS_Survivor_Space avgt 10 0.103 ± 0.477 MB/sec
FastAvroSerdesBenchmark.testFastAvroSerialization:·gc.churn.PS_Survivor_Space.norm avgt 10 0.852 ± 3.946 B/op
FastAvroSerdesBenchmark.testFastAvroSerialization:·gc.count avgt 10 8.000 counts
FastAvroSerdesBenchmark.testFastAvroSerialization:·gc.time avgt 10 36.000 ms
from avro-util.
That sounds great, but it is a separate improvement, which would like to have anyway.
But I think it is still important to understand the difference between 0.1.9 and the latest version by looking at the difference of the generated serializers.
from avro-util.
I have checked the generated serializers between 0.1.9 and the latest version under Avro 1.4. They are exactly the same BenchmarkSchemaGenericSerializer.
I found the old Avro-1.4 EnumArray benchmark result was not correct since the Enum value index used in regular avro-1.4 and fast-avro were different... The value string used in fast-avro benchmark happen to be shorter than the one used in regular avro-1.4. So fast-avro result was faster than regular avro 1.4 in the old test.
I re-benchmarked the EnumArray using 0.1.9 under Avro 1.4, the result was the same as above benchmark result. Fast-avro 0.1.9 is slower.
Benchmark Mode Cnt Score Error Units
FastAvroSerdesBenchmark.testFastAvroSerialization avgt 50 9855.346 ± 77.602 ns/op
from avro-util.
Related Issues (20)
- strict parsing shuold throw on extra content after end of json
- getNullableGenericDefaultValue API should fail Map type with null as default value.
- handle JsonNodes as arguments for default values in FieldBuilders
- support setting field props in schema field builders HOT 1
- Avro 1.11.1 incorrectly detected as 1.10 HOT 1
- Utils.inferCompileDependencies does not create a valid classpath on windows
- [helper] Add an API to return all Prop names.
- Consider adopting fastutil for deserialization of primitive collections
- Allow user-specified functional transforms to be applied during deserialization.
- commons-text CVE-2022-42889 HOT 1
- Fastavro code generator generates classes with String literals of length > 65K HOT 1
- make FieldBuilders validate that schema is != null on build()
- add assertCompiles() checks to codegen tests
- 0.2.143 not in Artifactory HOT 4
- Example usage of this util class HOT 1
- ArrayIndexOutofBoundsException when using avro-fastserde
- Deserialization error when using union for string type HOT 1
- LogicalTypes codegen issues HOT 1
- Class generated for schema with nested Maps is incorrect
- NPEs thrown by serialization for mandatory fields could have better error messages
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from avro-util.