aihuaxu / polaris Goto Github PK

Polaris is an open-source, fully-featured catalog for Apache Iceberg™. It implements Iceberg's REST API, enabling seamless multi-engine interoperability across a wide range of platforms, including Apache Doris™, Apache Flink®, Apache Spark™, StarRocks, and Trino.

Documentation is available at https://polaris.io, including Polaris management API doc and Apache Iceberg REST API doc.

Polaris Catalog is open source under an Apache 2.0 license.

• ⭐ Star this repo if you’d like to bookmark and come back to it!
• 📖 Read the announcement blog post for more details!

Polaris is organized into the following modules:

• polaris-core - The main Polaris entity definitions and core business logic
• polaris-server - The Polaris REST API server
• polaris-eclipselink - The Eclipselink implementation of the MetaStoreManager interface

Polaris is built using Gradle with Java 21+ and Docker 27+.

• ./gradlew build - To build and run tests. Make sure Docker is running, as the integration tests depend on it.
• ./gradlew assemble - To skip tests.
• ./gradlew test - To run unit tests and integration tests.
• ./gradlew runApp - To run the Polaris server locally on localhost:8181.
  • The server starts with the in-memory mode, and it prints the auto-generated credentials to STDOUT in a message like this realm: default-realm root principal credentials: <id>:<secret>
  • These credentials can be used as "Client ID" and "Client Secret" in OAuth2 requests (e.g. the curl command below).
• ./regtests/run.sh - To run regression tests or end-to-end tests in another terminal.

Running in Docker

• docker build -t localhost:5001/polaris:latest . - To build the image.
• docker run -p 8181:8181 localhost:5001/polaris:latest - To run the image in standalone mode.
• docker compose up --build --exit-code-from regtest - To run regression tests in a Docker environment.

Running in Kubernetes

• ./setup.sh - To run Polaris as a mini-deployment locally. This will create two pods that bind themselves to port 8181.
• kubectl get pods - To check the status of the pods.
• kubectl get deployment - To check the status of the deployment.
• kubectl describe deployment polaris-deployment - To troubleshoot if things aren't working as expected.

Building docs

• Docs are generated using Redocly. To regenerate them, run the following commands from the project root directory.

docker run -p 8080:80 -v ${PWD}:/spec docker.io/redocly/cli join spec/docs.yaml spec/polaris-management-service.yml spec/rest-catalog-open-api.yaml -o spec/index.yaml --prefix-components-with-info-prop title
docker run -p 8080:80 -v ${PWD}:/spec docker.io/redocly/cli build-docs spec/index.yaml --output=docs/index.html --config=spec/redocly.yaml

To connect from an engine like Spark, first create a catalog with these steps:

# Generate a token for the root principal, replacing <CLIENT_ID> and <CLIENT_SECRET> with
# the values from the Polaris server output.
export PRINCIPAL_TOKEN=$(curl -X POST http://localhost:8181/api/catalog/v1/oauth/tokens \
  -d 'grant_type=client_credentials&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>&scope=PRINCIPAL_ROLE:ALL' \
   | jq -r '.access_token')
   
# Create a catalog named `polaris`
curl -i -X POST -H "Authorization: Bearer $PRINCIPAL_TOKEN" -H 'Accept: application/json' -H 'Content-Type: application/json' \
  http://localhost:8181/api/management/v1/catalogs \
  -d '{"name": "polaris", "id": 100, "type": "INTERNAL", "readOnly": false, "storageConfigInfo": {"storageType": "FILE"}, "properties": {"default-base-location": "file:///tmp/polaris"}}'

From here, you can use Spark to create namespaces, tables, etc. More details can be found in the Quick Start Guide.

Apache Iceberg, Iceberg, Apache Spark, Spark, Apache Flink, Flink, Apache Doris, Doris, Apache, the Apache feather logo, the Apache Iceberg project logo, the Apache Spark project logo, the Apache Flink project logo, and the Apache Doris project logo are either registered trademarks or trademarks of The Apache Software Foundation. Copyright © 2024 The Apache Software Foundation.