The Barbarian Data System is the world's best cloud-first, cloud-agnostic data processing system founded on Apache Hadoop for enterprise-ready, low-latency parallel distributed data processing.

This release brings:

• Apache Hadoop 3.1.2
• Apache Hive 3.1, with LLAP & Tez
• Apache Zookeeper 3.5
• Apache Ignite 2.6

Barbarian is designed to run as a horizontally scalable parallel, distributed, in memory data warehouse - a bit like SAP Hana™ but with a low TCO.

Barbarian can ingest data from Amazon S3 or a remote HDFS cluster for high performance, low latency analysis, or alternatively Barbarian can be configured to run as a Big Data solution accessing Amazon S3 - using its own storage as an in-memory write-through cache. Lastly, Barbarian supports hybrid mode, whereby Barbarian mounts common paths in-memory and acts as a write-through cache of all other paths.

We will be adding support for more storage backends including Azure ADLS, GCP Cloud Storage, and Ceph in coming releases.

By default, Barbarian comes with Hive transactional table support, meaning inline record updates are possible for slowly changing dimensions and change-data-capture. You need to use the appropriate Hive syntax when creating a table for this to work:

CREATE TABLE mytable (id string, value1 string, value2 int)
CLUSTERED BY (id) INTO 5 BUCKETS
STORED AS ORC
TBLPROPERTIES ("transactional"="true");

More info at Apache https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions.

You can load data into Barbarian by declaring an external table over a directory in S3, like this:

CREATE EXTERNAL TABLE mys3table (id string, value1 string, value2 int)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION 's3a://my-bucket/my-prefix';
INSERT INTO mytable SELECT id, value1, value2 FROM mys3table;

Read more at: https://barbarians.io/, or read the docs at http://docs.barbarians.io/.

This repo contains the Helm charts for installing the Barbarian Data System on Kubernetes.

Install the Barbarian Data System with the following commands.

• helm repo add barbarians http://charts.barbarians.org/barbarian
• helm install --name my-barbarian barbarians/barbarian

To uninstall/delete the my-barbarian deployment:

• helm delete my-barbarian

The command removes all the Kubernetes components associated with the chart and deletes the release.

To quickly run a query from the shell, connect to the hiveserver2 node and fire up beeline:

HS2_HOST=$(kubectl get pods | grep hive-hs2 | awk '{ print $1 }' | tail -n 1)
kubectl exec -it $HS2_HOST /bin/bash
/opt/barbarian/hive/bin/beeline
!connect jdbc:hive2://localhost:10000
< hit enter twice >
set hive.execution.engine=tez;
set hive.llap.io.memory.mode=cache;
set hive.llap.execution.mode=all;
set hive.llap.io.enabled=true;
set hive.llap.daemon.service.hosts=@llap;
set hive.execution.mode=llap;
CREATE TABLE mytable (id string, value1 string, value2 int);
INSERT INTO mytable SELECT 'one', 'world', 1;
SELECT * FROM mytable;

Barbarian exposes many configuration parameters. Some important ones are listed below.

If you encounter provisioning issues with MariaDb where it gets stuck on "pending", this could be because you haven't enabled persistent storage claims. To enable persistent storage on AWS EKS you can run something like:

echo "\
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp2
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain
mountOptions:
  - debug
" > /tmp/storage-class.yaml

kubectl create -f /tmp/storage-class.yaml
kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'