opentsdb / asynccassandra Goto Github PK

A shim for using Cassandra as a backend for OpenTSDB. Not to be used as a general Cassandra client.

License: BSD 3-Clause "New" or "Revised" License

Java 100.00%

asynccassandra's Introduction

       ___                 _____ ____  ____  ____
      / _ \ _ __   ___ _ _|_   _/ ___||  _ \| __ )
     | | | | '_ \ / _ \ '_ \| | \___ \| | | |  _ \
     | |_| | |_) |  __/ | | | |  ___) | |_| | |_) |
      \___/| .__/ \___|_| |_|_| |____/|____/|____/
           |_|    The modern time series database.

OpenTSDB is a distributed, scalable Time Series Database (TSDB) written on
top of HBase.  OpenTSDB was written to address a common need: store, index
and serve metrics collected from computer systems (network gear, operating
systems, applications) at a large scale, and make this data easily accessible
and graphable.

Thanks to HBase's scalability, OpenTSDB allows you to collect thousands of
metrics from tens of thousands of hosts and applications, at a high rate
(every few seconds). OpenTSDB will never delete or downsample data and can
easily store hundreds of billions of data points.

OpenTSDB is free software and is available under both LGPLv2.1+ and GPLv3+.
Find more about OpenTSDB at http://opentsdb.net

asynccassandra's People

Contributors

Stargazers

Watchers

Forkers

manolama alienth pologood mbrukman lizunmvn paladintyrion tosoikea isabella232

asynccassandra's Issues

Config Cassendra DB

Where to add the Cassandra Db connectivity details ?
Present YML file have only asynccassandra.seeds where we are specifying only port and IP address of the Cassandra. Where to add the Cassendra DB user name and password details (Schema details)?

Build failure

Attempting to build the process by your docs I get the following error:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13.259 s
[INFO] Finished at: 2016-07-15T11:30:53-06:00
[INFO] Final Memory: 30M/271M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (attach-javadocs) on project asynccassandra: MavenReportException: Error while creating archive:
[ERROR] Exit code: 1 - /bin/sh: scl: line 1: syntax error: unexpected end of file
[ERROR] /bin/sh: error importing function definition for `BASH_FUNC_scl'

This happens when I just run mvn clean package in the root "asynccassandra" directory.

TTransportException: Frame size large than max lenght

Hello,

I'm running OpenTSDB 2.3 with Cassandra, and when query the metrics with many points, occurs the below exception. I changed the properties of Cassandra Server and not works. How to fix it?

Properties Cassandra Server:
thrift_framed_transport_size_in_mb: 60

The Exception:
Oct 25 16:52:04 opentsdb-qa2-01 tsdb: Caused by: org.apache.thrift.transport.TTransportException: Frame size (17572007) larger than max length (16384000)! Oct 25 16:52:04 opentsdb-qa2-01 tsdb: at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137) ~[asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar:na] Oct 25 16:52:04 opentsdb-qa2-01 tsdb: at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) ~[asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar:na] Oct 25 16:52:04 opentsdb-qa2-01 tsdb: at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) ~[asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar:na] Oct 25 16:52:04 opentsdb-qa2-01 tsdb: at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) ~[asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar:na] Oct 25 16:52:04 opentsdb-qa2-01 tsdb: at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) ~[asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar:na] Oct 25 16:52:04 opentsdb-qa2-01 tsdb: at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) ~[asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar:na] Oct 25 16:52:04 opentsdb-qa2-01 tsdb: at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) ~[asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar:na] Oct 25 16:52:04 opentsdb-qa2-01 tsdb: at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:811) ~[asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar:na] Oct 25 16:52:04 opentsdb-qa2-01 tsdb: at org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:795) ~[asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar:na] Oct 25 16:52:04 opentsdb-qa2-01 tsdb: at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$2$1.internalExecute(ThriftColumnFamilyQueryImpl.java:413) ~[asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar:na] Oct 25 16:52:04 opentsdb-qa2-01 tsdb: at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$2$1.internalExecute(ThriftColumnFamilyQueryImpl.java:400) ~[asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar:na] Oct 25 16:52:04 opentsdb-qa2-01 tsdb: at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60) ~[asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar:na] Oct 25 16:52:04 opentsdb-qa2-01 tsdb: ... 9 common frames omitted

ByteOrderedPartitioner

Hey,

when y try to run opentsdb with cassandra with ByteOrderedPartitioner it fails
without ByteOrderedPartitioner it succeed

what's wrong ?

:/usr/share/opentsdb# /usr/share/opentsdb/build/tsdb tsd --conf
ig=/etc/opentsdb/opentsdb.conf --port=4242 --staticroot=static/ --cachedir=/tmp
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/opentsdb/build/third_party/logback/
logback-classic-1.0.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/opentsdb/build/third_party/asynccas
sandra/asynccassandra-0.0.1-20160229.001338-4-jar-with-dependencies.jar!/org/slf
4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorSta
ticBinder]
2017-01-23 10:21:55,901 INFO  [main] TSDMain: Starting.
2017-01-23 10:21:55,904 INFO  [main] TSDMain: net.opentsdb.tools 2.3.0 built at
revision cac608a (MINT)
2017-01-23 10:21:55,904 INFO  [main] TSDMain: Built on 2017/01/20 14:16:41 +0000
 by root@33842653d6db:/usr/share/opentsdb/build
2017-01-23 10:21:55,906 INFO  [main] Config: Successfully loaded configuration f
ile: /etc/opentsdb/opentsdb.conf
2017-01-23 10:21:55,969 INFO  [main] Config: Successfully loaded configuration f
ile: /etc/opentsdb/opentsdb.conf
2017-01-23 10:21:56,031 WARN  [main] PluginLoader: Unable to locate any plugins
of the type: net.opentsdb.query.filter.TagVFilter
2017-01-23 10:21:56,096 INFO  [main] ConnectionPoolMBeanManager: Registering mbe
an: com.netflix.MonitoredResources:type=ASTYANAX,name=MyConnectionPool,ServiceTy
pe=connectionpool
2017-01-23 10:21:56,109 INFO  [main] CountingConnectionPoolMonitor: AddHost: cas
sandra
2017-01-23 10:21:56,179 INFO  [main] CountingConnectionPoolMonitor: AddHost: 10.
42.141.144
2017-01-23 10:21:56,179 INFO  [main] CountingConnectionPoolMonitor: RemoveHost:
cassandra
2017-01-23 10:21:56,195 INFO  [main] CountingConnectionPoolMonitor: AddHost: cas
sandra
2017-01-23 10:21:56,211 INFO  [main] CountingConnectionPoolMonitor: AddHost: 10.
42.141.144
2017-01-23 10:21:56,211 INFO  [main] CountingConnectionPoolMonitor: RemoveHost:
cassandra
2017-01-23 10:21:56,230 INFO  [main] TSDB: Flushing compaction queue
2017-01-23 10:21:56,235 ERROR [main] ConnectionPoolMBeanManager: com.netflix.Mon
itoredResources:type=ASTYANAX,name=MyConnectionPool,ServiceType=connectionpool
2017-01-23 10:21:56,236 INFO  [main] TSDB: Completed shutting down the TSDB
Exception in thread "main" java.lang.RuntimeException: Initialization failed
        at net.opentsdb.tools.TSDMain.main(TSDMain.java:237)
Caused by: com.stumbleupon.async.DeferredGroupException: At least one of the Def
erreds failed, first exception:
        at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169)
        at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.ja
va:142)
        at com.stumbleupon.async.DeferredGroup.access$000(DeferredGroup.java:36)
        at com.stumbleupon.async.DeferredGroup$1Notify.call(DeferredGroup.java:8
2)
        at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
        at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688)
        at com.stumbleupon.async.Deferred.addBoth(Deferred.java:772)
        at com.stumbleupon.async.DeferredGroup.<init>(DeferredGroup.java:119)
        at com.stumbleupon.async.Deferred.group(Deferred.java:897)
        at net.opentsdb.core.TSDB.checkNecessaryTablesExist(TSDB.java:593)
        at net.opentsdb.tools.TSDMain.main(TSDMain.java:195)
Caused by: java.util.concurrent.ExecutionException: com.netflix.astyanax.connect
ionpool.exceptions.NoAvailableHostsException: NoAvailableHostsException: [host=N
one(0.0.0.0):0, latency=0(0), attempts=0]No hosts to borrow from
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.hbase.async.HBaseClient$1ResponseCB.run(HBaseClient.java:170)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.netflix.astyanax.connectionpool.exceptions.NoAvailableHostsExcept
ion: NoAvailableHostsException: [host=None(0.0.0.0):0, latency=0(0), attempts=0]
No hosts to borrow from
        at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailove
r.<init>(RoundRobinExecuteWithFailover.java:48)
        at com.netflix.astyanax.connectionpool.impl.RoundRobinConnectionPoolImpl
.newExecuteWithFailover(RoundRobinConnectionPoolImpl.java:66)
        at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnect
ionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
        at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(Thr
iftColumnFamilyQueryImpl.java:186)
        at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$4.call(Thri
ftColumnFamilyQueryImpl.java:297)
        at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$4.call(Thri
ftColumnFamilyQueryImpl.java:294)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        ... 3 more

Thanks

asynccassandra for Cassandra 3x

Hello,
I'm trying to run Open TSDB with cassandra 3.7, but looks like it is not supported or not documented. For example i can't find cql script to create keyspace and column families - all that I can see in cassandra-cli script, which is moved out for cassandra 3x.

Can you please help with this issue?

Thanks!

building error

Removing the javadoc portion of the pom.xml
HBaseClient.java file missing import ConsistencyLevel

PoolTimeoutException with greater chunks of data

com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=10.1.103.80(10.1.103.80):9160, latency=10004(10004), attempts=1] Timed out waiting for connection at com.netflix.astyanax.th …

Facing issue when configuring opentsDB on Cassandra

root@raj-HP-EliteBook-8470p:/home/raj/Downloads/asynccassandra-master# mvn pack

[ERROR] /home/raj/Downloads/asynccassandra-master/src/org/hbase/async/HBaseClient.java:[366,35] error: cannot find symbol
[INFO] 5 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.173 s
[INFO] Finished at: 2018-04-03T15:15:39+05:30
[INFO] Final Memory: 26M/271M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project asynccassandra: Compilation failure: Compilation failure:
[ERROR] /home/raj/Downloads/asynccassandra-master/src/org/hbase/async/HBaseClient.java:[304,34] error: cannot find symbol
[ERROR] symbol: variable ConsistencyLevel
[ERROR] location: class HBaseClient
[ERROR] /home/raj/Downloads/asynccassandra-master/src/org/hbase/async/HBaseClient.java:[303,25] [deprecation] BoundedExponentialBackoff(long,int,int) in BoundedExponentialBackoff has been deprecated
[ERROR] /home/raj/Downloads/asynccassandra-master/src/org/hbase/async/HBaseClient.java:[311,35] error: cannot find symbol
[ERROR] symbol: variable ConsistencyLevel
[ERROR] location: class HBaseClient
[ERROR] /home/raj/Downloads/asynccassandra-master/src/org/hbase/async/HBaseClient.java:[353,25] [deprecation] BoundedExponentialBackoff(long,int,int) in BoundedExponentialBackoff has been deprecated
[ERROR] /home/raj/Downloads/asynccassandra-master/src/org/hbase/async/HBaseClient.java:[366,35] error: cannot find symbol
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Under moderate load, JVM eventually spins indefinitely when this shim is in use

After writing a few hundred thousand datapoints, the JVM inevitably eats up all CPU and sits there not doing much. Even if you stop writing datapoints to the service, the process will sit there indefinitely eating up CPU. JVM becomes unresponsive to sleuthing via JMX or jvmtop. Tried both OpenJDK and Oracle JRE, both 1.8, and got the same behaviour. Very reproducable: Jam hundreds of thousands of datapoints at it for ~10 or so minutes.

When the proc enters this state, no communication is happening with Cassandra.

All of the metrics and tagk/tagvs I'm sending already have IDs, so I don't believe it is an issue with the shim's pseudo row lock method to acquire those.

I was able to get two thread dumps while a proc was in this state. These dumps are on the same PID, separated by a few minutes.

threads5.txt
threads6.txt

Some type of soft resource deadlock, perhaps?

PoolTimeoutException:

We are getting below exception while running OpenTSDB .
2016-11-14 04:10:00,813 ERROR [main] ConnectionPoolMBeanManager: com.netflix.MonitoredResources:type=ASTYANAX,name=MyConnectionPool,ServiceType=connectionpool
2016-11-14 04:10:00,813 INFO [main] TSDB: Completed shutting down the TSDB
Exception in thread "main" java.lang.RuntimeException: Initialization failed
at net.opentsdb.tools.TSDMain.main(TSDMain.java:233)
Caused by: com.stumbleupon.async.DeferredGroupException: At least one of the Deferreds failed, first exception:
at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169)
at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:142)
at com.stumbleupon.async.DeferredGroup.access$000(DeferredGroup.java:36)
at com.stumbleupon.async.DeferredGroup$1Notify.call(DeferredGroup.java:82)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1005)
at org.hbase.async.HBaseClient$1ResponseCB.run(HBaseClient.java:185)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=127.0.0.1(127.0.0.1):9160, latency=2004(2004), attempts=1]Timed out waiting for connection
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.hbase.async.HBaseClient$1ResponseCB.run(HBaseClient.java:170)
... 3 more
Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=127.0.0.1(127.0.0.1):9160, latency=2004(2004), attempts=1]Timed out waiting for connection
at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:231)
at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:198)
at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:84)
at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:117)
at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:186)
at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$4.call(ThriftColumnFamilyQueryImpl.java:297)
at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$4.call(ThriftColumnFamilyQueryImpl.java:294)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more