What would you like to be improved?

Usually, a successful project should have an outstanding name in the beginning:

1. it should be easy to be discovered (like: search over web) and remember, so usually to be one word (not words combination) and not the word that rarely in use.
2. it should be meaningful and usually can hint what project's intention to do.
3. it won't conflict with some well-known projects, such as hadoop, spark, flink, kubernetes, etc.

How should we improve?

No response

Leave detailed comments to be added by assignee.
Also detailed design doc is required.

Describe the subtask

This issue tracks the work of implementing the maintain the tables in the hive Using namespace SPI.

Parent issue

Describe the subtask

This issue tracks the work of implementing a client interface to manipulate lakehouses.

Parent issue

#63

What would you like to be improved?

Currently we only implemented the http ServerConnect for embedded jetty server, it would be better to add the https support for security and completeness.

How should we improve?

Simply adding Jetty https ServerConnect should be enough.

Describe the proposal

This issue tracks the works of design and implements a geo-distributed HA architecture for Graviton. The owner of this epic should have a design in detail and break down into subtasks.

Task list

No response

Describe the proposal

This issues tracks the work of extending the catalog ability to support more heterogeneous catalog, like file catalog, ML catalog, and so on. The epic task should have a design in detail and break down to subtasks.

Task list

• #9

Describe the proposal

The hive metadata connector is the hive data source core system in Graviton to describe how the connector creates and runs.

This umbrella issue tracks the whole design and implementation of Graviton's hive metadata connector(not data connector) system, including:

1. Maintain the database in the hive Using namespace SPI.
2. Maintain the table in the hive Using namespace SPI.
3. Waiting to be added

Task list

• #59
• #87
• #61
• #248
• #90
• #91
• #196

Leave detailed comments to be added by assignee.
Note. This is an umbrella issue, needs to break into sub-issues.
Also detailed design doc is required.

Describe the subtask

This issue tracks the work of implementing Catalog REST operations for Graviton Server.

Parent issue

#6

Describe the subtask

This task aims to implement the on-wire protocol for schema system. As specified in the design doc, we will choose to use JSON format for the on-wire protocol and implement the JSON serialization/deserialization for schema system.

Parent issue

#3

Describe the subtask

This task tracks the work of updating the RFC-1 schema spec to adapt to the new design.

Parent issue

#43

Describe the proposal

This epic tracks the work about adapting Graviton catalog's ability to the specific query engine. With this, query engine could use Graviton to manage the metadata from underlying system.

Task list

• #56
• #57
• #244

This issue tracks the design of metadata spec, including memory structure, on-wire protocol and storage layout.

This is the subtask of parent issue #3

This epic tracks the work of designing metadata entity storage interface and achieve one implementation.

The works are:

• #48
• #50
• #51
• #112

Describe the feature

This issue aims to design and implement the new interfaces for metadata manipulating, including GET, UPDATE, CREATE and DELETE of metadata.

1. The issue and related PR only focus on the implementation of interfaces, not the specific implementations.
2. The interfaces are used internally, and will not expose to the end users.

Motivation

The implementation of metadata manipulation interfaces decouples the interface from specific implementation. And this will be used by Graviton Server.

Describe the solution

No response

Additional context

No response

Describe the subtask

This issue tracks the work of implementing metadata and type spec (specifically memory structure of metadata schema).

Parent issue

#3

Describe the subtask

This issue tracks the works of implementing a client Catalog manipulation interface for Graviton.

Parent issue

#57

What would you like to be improved?

Currently dependencies are defined across the build files, which makes dependency management quite mess, we should refactor the current build file to manage all the dependencies in a single place.

How should we improve?

One solution is to introduce version lock mechanism like Iceberg, but the palantir version lock has lots of limitations, one of the big problems is that it only supports one version for one dependency, it is hard to extend to support multi-versions in one project.

Another solution is to introduce spring plugin to manage dependency versions in one place.

Describe the subtask

No response

Parent issue

#7

Describe the proposal

Current Graviton's Schema system follows "lakehouse/zone/table" logical structure, while the mapping physical table has different physical structure "metasouce/db/table", it's quite confuse for user to understand these two metadata structures, also it's hard to manage the metadata. So we decide to redesign the schema system to make it simpler and easy to understand.

The details of thinking can be found here (https://docs.google.com/document/d/1Wrd9HHJF2wLDhsvKix1VVKWLi_RdDhw-X3rqg-4lwtw/edit#heading=h.4adypg9kzjky).

Task list

• #44
• #46

Describe the subtask

This task aims to implement the Metadate Connector plugin Framework. As specified in the design doc, we will implement a plug-in factory, and let every metadata connector type be created and run through it.

Parent issue

#7

What would you like to be improved?

Currently, we are using terminolog Lakehouse to represent the top-level repository for metadata management. The fact is that we just manages the metadata, so it is not so accurate to use Lakehouse, instead I propose Metalake to represent the metadata lake.

How should we improve?

No response

Describe the subtask

This issue tracks the work of designing the client API interface for Graviton. The client API is mainly used for the users to interact with Graviton to manipulate metadata, also this client API is the common client that will suit to query engines like Spark, Trino later on.

Parent issue

#53

This epic issue aims to design and implement the REST API framework for Graviton.

1. The REST API design follows Microsoft's REST API design specs (https://learn.microsoft.com/en-us/azure/architecture/best-practices/api-design#define-api-operations-in-terms-of-http-methods).
2. Besides, we use MediaType's versioning mechanism to control the API version.
3. We use Jersey for REST API implementation. The purpose of using Jersey is that Jersey is much light-weighted than Springboot, and we don't want things out of control with so many injection mechanisms introduced by Springboot.

Subtasks are:

• #34
• #35
• #36
• #39
• #72
• #88
• #144

Describe the subtask

This issue tracks the work of implementing the maintain the database in the hive Using namespace SPI.

Parent issue

#58

Describe the subtask

This issue tracks the work of introducing Jersey test framework for REST APIs, so that we could verify in UTs.

Parent issue

#6

AWS has Graviton. https://aws.amazon.com/ec2/graviton/ and have registered the "AWS Graviton" trademark in multiple countries. Class 42 trademarks include software and hardware. These people also exist https://www.gravitonusa.com. This might become an issue upon entry into the ASF Incubator due to those existing names. Do we want to keep the name or come up with something else?

What would you like to be improved?

During implementing HiveCatalog, we unexpectedly introduce so many unnecessary transient dependencies, which will easily cause conflicts and increase the package size. We should clean all of them.

How should we improve?

Exclude all the unnecessary dependencies.

Substrait simple type BINARY not found in schema.proto, do we need to add it?

Describe the subtask

This issue tracks the design of Metadata Connector, including the blow part :

• Connector plugin run mode

Parent issue

#7

The metadata connector system is the core system in Graviton to describe the connector how to create, and run, including SPI(service provider interface), different data source's field type mapping, and memory structure.

This umbrella issue tracks the whole design and implementation of Graviton's metadata connector(not data connector) system, including:

1. Connector factory design & implementation.
2. Connector extract metadata job schedule design & implementation.
3. Connector plug-in framework.
4. Unified metadata type definite.
5. Type converter design & implementation.
6. Metadata multiple version design & implementation.
7. Typical connector design & implementation.

Subtasks are list in here:

• #19
• #28

What would you like to be improved?

To improve the security and avoid attacks to our embedded server, we should add cors filter support for our Jetty server.

How should we improve?

Implement a cors filter and register into Jetty should be enough.

Describe the subtask

This issue aims to track the work of implementing REST APIs for "Zone" CREATE/GET/UPDATE/DELETE operations.

Parent issue

#6

Leave detailed comments to be added by assignee.

Describe the subtask

This issue tracks the work of designing a metadata storage interface for Graviton.

• The interface should support storing entities to the underlying system.
• The interface should be generic enough to support different underlying storage (like kv store, relational DB, or others).

Parent issue

#4

To donate a project to the ASF we either need a Software grant and ICLA from contributors who don't work for Datastrato or ICLAs from everyone. Setting up a manual or automated ICLA system would make the donation process easier.

Suggested ICLA text based on v2.2 of the ASF's ICLA documents - I changed Foundation to company and removed mention of non-profit and CCLAs. I also simplified the form a little and added a GitHub id as sometimes matching GitHub names to real names also causes issues. We can start with a manual process first.

Datastrato Individual Contribitor License Agremment

This document is based on the Apache ICLA V2.2.

Thank you for your interest in Datastrato. (the "Company"). To clarify the intellectual property license granted with Contributions from any person or entity, the Company must have on file a signed Individual Contributor License Agreement ("ICLA") from each Contributor, indicating agreement with the license terms below. This agreement is for your protection as a Contributor as well as the protection of the Company and its users. It does not change your rights to use your own Contributions for any other purpose.

Please complete and sign this Agreement, and then email a copy to [email protected].

Read this document carefully before signing and keep a copy for your records.
GitHub Name: ____________________________________________________
Full Name: ______________________________________________________
Postal Address: _________________________________________________
_________________________________________________
Country: _________________________________________________
E-Mail: ______________________________________________________

You accept and agree to the following terms and conditions for Your Contributions (present and future) that you submit to the Company. In return, the Company shall not use Your Contributions in a way that is contrary to open source in effect at the time of the Contribution. Except for the license granted herein to the Company and recipients of software distributed by the Company, You reserve all right, title, and interest in and to Your Contributions.

1. Definitions.
   "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with the Company. For legal entities, the entity making a contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to the Company for inclusion in, or documentation of, any of the products owned or managed by the Company (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Company or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Company for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."

2. Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to the Company and to recipients of software distributed by the Company a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works.

3. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to the Company and to recipients of software distributed by the Company a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.

4. You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, you represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to the Company.

5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions.

6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.

7. Should You wish to submit work that is not Your original creation, You may submit it to the Company separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]".

8. You agree to notify the Company of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect.

Please sign: __________________________________ Date: ________________

Describe the subtask

This issue tracks the work of implementing a REST client for Graviton client. With this REST client, Graviton client could issue request to Graviton server to manipulate metadata.

Parent issue

#57

Describe the subtask

This issue tracks the work of implementing the create hive client in the HiveCatalog.

• This hive client is created by reflection and can support hive2 and hive3.

Parent issue

Describe the subtask

This issue is a follow-up issue of #48 , to implement one underlying storage for Graviton.

A simple candidate would be rocksdb, or we could choose FDB or other distributed storage for performance and fault tolerance.

Parent issue

#4

Task list

• #122
• #121
• #120

Describe the subtask

This subtask aims to create Tenant operation( including CREATE, GET, and DELETE) REST APIs for Graviton.

Parent issue

#6

Describe the subtask

This task aims to add support of protobuf serialization and deserialization support for schema system. With this system could persist and schema to storage, or communicate between servers using grpc.

Parent issue

#3

Describe the subtask

This issue tracks the work of implementing the common client for Graviton. With this client, users could manipulate metadata from Graviton to the underlying systems.

Parent issue

#53

Subtasks

• #64
• #66
• #71
• #145
• #146

Describe the subtask

This issue aims to track the work of designing entity serde interface for Graviton storage system.

The serde interface and the implementation will be used by the EntityStore module to serialize and deserialize the entity when interacting with the underlying storage system.

Parent issue

#4

Describe the subtask

This issue tracks to refactor the schema system:

1. Simplify the current "tenant/lakehouse/zone/table" logical structure.
2. reorganize the project structure.
3. Abstract the common entities and make them inheritable.

Parent issue

#43

Describe the feature

This task aims to add config system for the project.

Motivation

Config system is the cornerstone of the project, we should introduce a good Config system.

Describe the solution

No response

Additional context

No response

Describe the proposal

This epic tracks the work of achieving the packaging and launching process for Graviton. This includes:

1. Define and implement the process to package the project.
2. Define and implement the process to publish the project.
3. Add scripts and folders for launching GravitonServer.
4. Add docker files for GravitonServer (nice to have).

Task list

• #150
• #175
• #178
• #177

MVN Central Repository

• https://issues.sonatype.org/browse/OSSRH-93806

Describe the subtask

This issue aims to track the work of implementing REST APIs for "Lakehouse" CREATE/GET/UPDATE/DELETE operations.

Parent issue

#6

Metadata schema system is the core system in Unified Catalog to describe the entities, including memory structure, on-wire format and storage layout.

This umbrella issue tracks the whole design and implementation of Unified Catalog' schema system, including:

1. Metadata entity design & implement.
2. Entity serialization and deserialization protocol design & implement.
3. Entity storage layout design & implement.

Subtasks are listed here:

• Metadata and Type Spec design ( lakehouse, zone, table, column and other basic entities), including memory structure, on-wire and storage layout design. #11
• Metadata and Type Spec implementation (memory structure). #12
• Metadata on-wire protocol implement. #16
• Medata storage layout implement. #21