concaf / data-model Goto Github PK

Gremlin+Titan+DynamoDB is able to support multiple separate graphs in a single DynamoDB endpoint. For that we need to be able to change value on line 19 in dynamodb.properties:

storage.dynamodb.prefix=v100

It would probably make sense to make this configurable through ENV variable

Not an issue, just FYI: I moved gremlin bits to https://github.com/fabric8-analytics/gremlin-docker.

The reason for the move is that gremlin image is still under heavy development (we change the image a lot) and therefore the project is not a good fit for the CCCP (IMO). Building such still-WIP image in CCCP and relying on openshift/saas for prod deployments is quite painful experience.

Having the gremlin bits in fabric8-analytics/gremlin-docker will allow us to change and test the image more quickly. Once the image is truly stable, we can utilize CCCP again.

Thanks everyone who contributed and helped with the image here! :)

We need to create and enable CI jobs.

dynamodb.properties file gives the flexibility of configuring read and write units. Please make it configurable with a default value of 25.

1. Make following values in dynamodb.properties file
   storage.dynamodb.stores.edgestore.capacity-read=25
   storage.dynamodb.stores.edgestore.capacity-write=25
   storage.dynamodb.stores.graphindex.capacity-read=25
   storage.dynamodb.stores.graphindex.capacity-write=25

2. If an environment variable is provided, use that value instead of above default ones.

https://github.com/containscafeine/data-model/blob/master/gremlin/scripts/entrypoint.sh#L39

The above line contains a sleep 20. We can remove it.

The gremlin image is quite big (2.3 GiB). It contains leftovers from Maven build:

# du -hs /root/.m2/
123M	/root/.m2/

There are also other files of significant size in /opt/ that could be removed.

Currently, we have Xms and Xmx settings of 2 GB. I feel it should be a good idea to make it configurable so that based on the type of image (WS or HTTP) we could handle the memory requirements separately.

After the recent pull request that was merged : #11

I can't figure out if the image was actually built. How can this be resolved?

I suppose a Github integration with Container Pipeline should be enough -- something that marks the github repo with an Image build update message.

Building the gremlin container fails with the following:

 ---> 93f3674e1df7
Removing intermediate container 109d19efd167
Step 7/14 : RUN git clone https://github.com/awslabs/dynamodb-titan-storage-backend.git /opt/dynamodb/dynamodb-titan-storage-backend/ &&    cd /opt/dynamodb/dynamodb-titan-storage-backend/ &&    curl https://gist.githubusercontent.com/pluradj/d56c1948f4665ee7fb1bc35daeba4f92/raw/be5f639a64c8d6ac196c59eb7e6d1a1903015b17/dynamo-titan11-tp323.patch | git apply -v --index &&    mvn install
 ---> Running in 36158226c145
Cloning into '/opt/dynamodb/dynamodb-titan-storage-backend'...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5725  100  5725    0     0   4140      0  0:00:01  0:00:01 --:--:--  4142
Checking patch pom.xml...
error: while searching for:
        <dynamodb-local.endpoint>http://localhost:${dynamodb-local.port}</dynamodb-local.endpoint>
        <jdk.version>1.8</jdk.version>
        <aws.java.sdk.version>[1.10.5.1, 2.0.0)</aws.java.sdk.version>
        <titan.version>1.0.0</titan.version>
        <tinkerpop.version>3.0.1-incubating</tinkerpop.version>
        <dependency.plugin.version>2.10</dependency.plugin.version>
        <maven.assembly.plugin.version>2.5.5</maven.assembly.plugin.version>
        <maven.compiler.plugin.version>3.3</maven.compiler.plugin.version>

error: patch failed: pom.xml:20
error: pom.xml: patch does not apply
Checking patch src/test/resources/gremlin-server-local.yaml...
Checking patch src/test/resources/gremlin-server.yaml...
Checking patch src/test/resources/install-gremlin-server.sh...
The command '/bin/sh -c git clone https://github.com/awslabs/dynamodb-titan-storage-backend.git /opt/dynamodb/dynamodb-titan-storage-backend/ &&    cd /opt/dynamodb/dynamodb-titan-storage-backend/ &&    curl https://gist.githubusercontent.com/pluradj/d56c1948f4665ee7fb1bc35daeba4f92/raw/be5f639a64c8d6ac196c59eb7e6d1a1903015b17/dynamo-titan11-tp323.patch | git apply -v --index &&    mvn install' returned a non-zero code: 1

@miteshvp @vpavlin PTAL!

We need to be able to enable metrics (or keep them enabled by default) as described in

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.TitanDB.BestPractices.html#Tools.TitanDB.Metrics

here

https://github.com/awslabs/dynamodb-titan-storage-backend/blob/1.0.0/src/test/resources/dynamodb.properties#L6