Provide more consumable way to use AVSC files rather than JARs about avro-schema-examples HOT 8 CLOSED

zolyfarkas commented on May 24, 2024

Provide more consumable way to use AVSC files rather than JARs

from avro-schema-examples.

Comments (8)

zolyfarkas commented on May 24, 2024 1

I have made a java schema schema client implementation for maven repos, available at: https://github.com/zolyfarkas/spf4j-jaxrs/blob/master/spf4j-jaxrs-client/src/main/java/org/spf4j/avro/SchemaClient.java

from avro-schema-examples.

zolyfarkas commented on May 24, 2024

Jordan, jar’s are zip files and they can easily be opened in any language… however the publishing is configurable and one can publish the file as a zip. You are right, you have to unzip a file, but I don’t see why that is an issue, it is done once, and cached locally by a “schema client”… (you can use the maven client to do that transparently) There are maven repositories that can serve zip content like: http://repo/group/artifact/version/artifact.ziup!path/to/schema.asvc <http://repo/group/artifact/version/artifact.ziup!path/to/schema.asvc> Compatibility (and more) is checked during the build time in my examples, you have full control how far behind you want to validate using maven version range expression… like [1.0, ) I am aware of the confluent schema repository, however here is where maven approach is “better“: 1) By using maven, you really take advantage of a piece of architecture that you already have, one less moving part. Simple. 2) You can use the maven dependency system to compose/reuse schemas, like: https://github.com/zolyfarkas/avro-schema-examples/tree/master/test-schema <https://github.com/zolyfarkas/avro-schema-examples/tree/master/test-schema> depends on https://github.com/zolyfarkas/avro-schema-examples/tree/master/test-common <https://github.com/zolyfarkas/avro-schema-examples/tree/master/test-common> 3) You can publish html avrodoc… same pattern as with javadoc see <https://bintray.com/zolyfarkas/core/download_file?file_path=org%2Fspf4j%2Favro%2Fexamples%2Ftest-schema%2F1.1%2Ftest-schema-1.1-avrodoc.jar> 4) Any checks you can imagine, can be run as part of the build, like in the example: compatibility, documentation, naming conventions…. 5) along with schema, you publish generated java, c#, typescript objects in their respective repositories (maven, nugget, npm). no need for devs to generate code again... Our schema development is same as code development… in git, reviewed, validated, published… and we have thousands of schemas… devs are not used to generate languages bindings anymore… it all already done as part of the schema build process…. cheers —Z

…

On Dec 15, 2018, at 1:06 PM, Jordan Moore ***@***.***> wrote: Avro supports more than just Java, so while packaging generated Java classes into a JAR is useful, putting the AVSC files into one, I feel, is not. $ jar -tf test-schema-1.1-avsc.jar org/spf4j/base/avro/JPackageInfo.avsc org/spf4j/base/avro/JThrowable.avsc org/spf4j/base/avro/TestRecord.avsc org/spf4j/base/avro/JFileLocation.avsc org/spf4j/base/avro/JStackTraceElement.avsc org/spf4j/base/avro/LocalKeyValue.avsc org/apache/avro/logicalTypes/AnyRecord.avsc codegen.properties schema_index.properties Therefore, your comment on the Avro JIRA about "using a maven repo as a schema repo REST service" I feel doesn't quite hold until one unzips the JAR file. Compare this to using the Confluent Schema Registry Maven Plugin <https://docs.confluent.io/current/schema-registry/docs/maven-plugin.html> which will publish the AVSC to a versioned, compatibility checked location that only requires an HTTP call to publish/receive. If you would like to contribute/browse code of that project, you can find it here - https://github.com/confluentinc/schema-registry/tree/master/maven-plugin <https://github.com/confluentinc/schema-registry/tree/master/maven-plugin> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIy1cH9W4t2rTSPXSgtpgod0wwUcIMlks5u5To-gaJpZM4ZU1oJ>.

from avro-schema-examples.

OneCricketeer commented on May 24, 2024

have to unzip a file, but I don’t see why that is an issue

Let's say I am using an embedded device, have little resources, and would like a single AVSC? What is the proposed solution there?
I guess my point is that if deploying to an HTTP file server, maybe should be configurable/shown how to deploy just the individual AVSC files and standalone artifacts rather than bundling them all. Of course, if you have imports between schemas, then multiples will need gotten, so in that case, the JAR/ZIP would be better... I can see it either way, but I find most developers I know wouldn't immediately be comfortable downloading an extracting contents of a JAR/ZIP within their code - and might not even understand a JAR is a ZIP-compatible format. If you can have http://.../artifact.zip!schema.avsc, then that seems to be a happy medium to me.

I completely am for using Maven and storing the schema in VCS, and even doing it in other repos, and others are doing it as well, such as Divolte Collector for doing web-activity tracking through Avro onto Kafka + Hadoop.

I'm not saying the Schema Registry is the alternative, or "superior" to this approach. If one needs to use the Confluent Maven Plugin on top of this, then that's their prerogative. My point was simply that, from a consumption perspective, being able to pin-point a specific Schema rather than the whole bunch, seems useful, and requires less resources in the end.

While I also agree with all the other points, could you help me understand 5? How would using Maven publish C# or TypeScript sources?

from avro-schema-examples.

zolyfarkas commented on May 24, 2024

We are running artifactory (from JFrog) as a maven repo, and artifactory allows you to address zip contents directly with the syntax: http://repo/group/artifact.zip!individual.avsc <http://repo/group/artifact.zip!individual.avsc> I am not sure why bintray (where I publish my examples) which is also a JFrog product does not do that…. on point 5) as part of the build process, we use a maven plugin (to be open sourced someday) that will run the code generation for C#, Typescript, Python, and will publish it to their respective repositories: nugget, npm, pypi. These repositories are not maven repositories, but in our case are hosted by artifactory just like maven is… Now this solution works really well, and we built our micro service (REST) , and messaging architecture on top of this approach. We use Content-Schema-Ref HTTP header to reference the content schema. We also have Content-Schema to put the entire schema in the http header, and we do that for SNAPSHOT and projections(partial schemas) A large percentage of client server interaction, uses partial objects (projections), and for those we currently use Accept-Content-Schema (client tells the server what part he needs), and Content-Schema for describing the payload. These 2 headers contain the entire schemas (stripped from doc fields and other serialization unrelated fields). Now for these more dynamic use cases, one could use a more dynamic schema repo than maven, to remove the wire overhead… especially that HTTP headers have size limitations… for example a User selects 8 fields out of 100 available in a UI, client published this partial schema to a schema repo, gets an ID back, and this is what it will used to identify the projection… instead of currently sending the entire schema... —Z

…

On Dec 15, 2018, at 6:29 PM, Jordan Moore ***@***.***> wrote: have to unzip a file, but I don’t see why that is an issue Let's say I am using an embedded device, have little resources, and would like a single AVSC? What is the proposed solution there? I guess my point is that if deploying to an HTTP file server, maybe should be configurable/shown how to deploy just the individual AVSC files and standalone artifacts rather than bundling them all. Of course, if you have imports between schemas, then multiples will need gotten, so in that case, the JAR/ZIP would be better... I can see it either way, but I find most developers I know wouldn't immediately be comfortable downloading an extracting contents of a JAR/ZIP within their code - and might not even understand a JAR is a ZIP-compatible format. If you can have http://.../artifact.zip!schema.avsc, then that seems to be a happy medium to me. I completely am for using Maven and storing the schema in VCS, and even doing it in other repos, and others are doing it as well, such as Divolte Collector <https://github.com/divolte/divolte-schema> for doing web-activity tracking through Avro onto Kafka + Hadoop. I'm not saying the Schema Registry is the alternative, or "superior" to this approach. If one needs to use the Confluent Maven Plugin on top of this, then that's their prerogative. My point was simply that, from a consumption perspective, being able to pin-point a specific Schema rather than the whole bunch, seems useful, and requires less resources in the end. While I also agree with all the other points, could you help me understand 5? How would using Maven publish C# or TypeScript sources? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIy1ckJzvdhcLIbi91rEb0ymwIwDpONks5u5YXpgaJpZM4ZU1oJ>.

from avro-schema-examples.

OneCricketeer commented on May 24, 2024

for example a User selects 8 fields out of 100 available in a UI, client published this partial schema to a schema repo, gets an ID back, and this is what it will used to identify the projection… instead of currently sending the entire schema...

Sounds like something where GraphQL could help

from avro-schema-examples.

zolyfarkas commented on May 24, 2024

Indeed, I have been looking at it… will need to do a lot of work to integrate it with avro though since JSON is not feasible for a large number of services… Early days though, we currently have projections working without GraphQL… but will definitely look at evolving this into a more “mainstream” solution... I see more potential for GraphQL for federating services (like https://nordicapis.com/federated-data-with-hypergraphql/ <https://nordicapis.com/federated-data-with-hypergraphql/> ) —Z

…

On Dec 16, 2018, at 6:18 PM, Jordan Moore ***@***.***> wrote: for example a User selects 8 fields out of 100 available in a UI, client published this partial schema to a schema repo, gets an ID back, and this is what it will used to identify the projection… instead of currently sending the entire schema... Sounds like something where GraphQL could help — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIy1esYc_s70PeheYUn_1Pnc_-1zZTUks5u5tTQgaJpZM4ZU1oJ>.

from avro-schema-examples.

zolyfarkas commented on May 24, 2024

@OneCricketeer regarding GraphQL, take a look at: https://github.com/zolyfarkas/jaxrs-spf4j-demo/wiki/AvroCalciteRest.

Along with things (avro schema references) built on top of this approach:

https://github.com/zolyfarkas/jaxrs-spf4j-demo/wiki

cheers

from avro-schema-examples.

OneCricketeer commented on May 24, 2024

👀Thanks for the update

from avro-schema-examples.

Provide more consumable way to use AVSC files rather than JARs about avro-schema-examples HOT 8 CLOSED

Comments (8)

Related Issues (2)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent