thelastpickle / cassandra-reaper Goto Github PK

Automated Repair Awesomeness for Apache Cassandra

License: Apache License 2.0

Makefile 0.14% Shell 2.07% Python 1.61% Java 68.45% HTML 0.04% JavaScript 14.60% Gherkin 2.41% Dockerfile 0.38% SCSS 0.14% Less 10.09% EJS 0.07%

apache-cassandra cassandra repairs cassandra-repairs repair-schedules cassandra-clusters compactions snapshots cleanups

cassandra-reaper's Introduction

Reaper for Apache Cassandra

Reaper is a centralized, stateful, and highly configurable tool for running Apache Cassandra repairs against single or multi-site clusters.

The current version supports running Apache Cassandra cluster repairs in a segmented manner, opportunistically running multiple parallel repairs at the same time on different nodes within the cluster. Basic repair scheduling functionality is also supported.

Reaper comes with a GUI, which if you're running in local mode can be at http://localhost:8080/webui/

Please see the Issues section for more information on planned development, and known issues.

Documentation and Help

The full documentation is available at the Reaper website. The source for the site is located in this repo at src/docs.

Have a question? Join us on the ASF Slack in the #cassandra-reaper channel.

System Overview

Reaper consists of a database containing the full state of the system, a REST-full API, and a CLI tool called spreaper that provides an alternative way to issue commands to a running Reaper instance. Communication with Cassandra nodes in registered clusters is handled through JMX.

Reaper system does not use internal caches for state changes regarding running repairs and registered clusters, which means that any changes done to the storage will reflect to the running system dynamically.

You can also run the Reaper with memory storage, which is not persistent, and is meant to be used only for testing purposes.

This project is built on top of Dropwizard: http://dropwizard.io/

Version compatibility

Reaper can be built using Java 8 or 11. It is tested against Cassandra 3.11 and 4.0. It is no longer tested against Cassandra 2.x.

We have confirmed the Reaper UI will build with npm 5.6.0, node 10.0.0. We believe that more generally versions of npm up to 6.14 and both node 12.x and 14.x will work. Builds are confirmed to fail with node 16+.

We recommend the use of nvm to manage node versions.

Dependencies

Reaper uses an unmodified EPL-2.0 licensed dependency: EclipseStore. The source code can be found in the GitHub repository.

Note: This repo is a fork from the original Reaper project, created by the awesome folks at Spotify.

cassandra-reaper's People

Contributors

Stargazers

Watchers

Forkers

vinaykumarchella sarohaatul nisc-acooper beckje01 llinder eslocombe neogenix mgobec lqid adejanovski slivne spodkowinski segfault mstump slitsche reddit superpaul akonkol epfremmer chrislbs skhatri bj0rnen sgpappu rrs0108 patrickclee0207 anguenot jsanda jagodevreede joeceramic favoretti agoetz00 smarsching misterserious gitter-badger mrwulf elsmorian bvysocky znick bhumit velo vickyatp internetstaff vidigalp vdemonchy ymclinux kipwoker vektorich signalfx tedoc2000 shendrickx jkondapalli03 debswade mehrdad-shokri wloczykij2 marshallking kingm5 sunkarasreenu dormidon kousin197804 diegopacheco bakink jalaziz terrcin fosholaw pi4alina benbromhead a1exsh danielkza pndewit denniskline cognicraftio jxie418 adamhutson exoschubes sgml arastoul gisripa rborer krisbushover dynanets paddybyers bgloeckle sebbonnet dkirkpatrick7 chakra-coder metabol samuelbenhamou bamartos fmarslan pc-jedi viossat b13n1u saleil-bhat aahutsal mmonemali yusufaytas smiklosovic alokamvenki402 linpower jsevellec

cassandra-reaper's Issues

Repair Doesnot start

When I execute the repair through reaper for a particular keyspace.The repair does not start and I can find
Info:Starting a run with id #11 with current state 'NOT_STARTED'
Info:scheduling repair for repair run #11

Could anyone please help me with this issue

support for repairing all keyspaces

Hello,

Currently when scheduling or running a repair there is a filed for keyspace, but is it possible to have a wild card * so that all the keyspaces will be repaired.

In my usecase we have multiple keyspaces that gets generated based on the req, it is very much impossible to use keyspace name to repair

Per DC reaper support

In a multi dc setup, you should be able to configure a reaper per dc. This feature would require using Cassandra storage and would involve coordination between DCs.

Formalize branching and release cycle conventions

This can probably be hashed out pretty quickly, but I want it here as a reminder. I want to make sure we adopt a formal release process starting with the 0.4 release in order to be able to make bug fixes as well as adopt new features.

My proposal is to follow semver closely:

For feature development:

New features is done in branches off master
Features are to be reviewed by a committer
New features are merged to master

For a release:

1a. For a minor release (0.4, 0.5, 1.0, etc) start by cutting a new branch. We can bikeshed the naming convention but for now I'll just use x.y. No patch indicated. Any preference on a prefix?
1b. For a bug fix release cutting a new branch is not necessary, as the bug fix should be merged into the branch created in 1a.
2. Create a new tag for the release prefixed with v plus the semver release version. For example, v1.0.0 is a major release, v0.4.0 is a minor release v0.4.1 is a bug fix release.
3. Major releases are done to indicate backwards incompatibility. The exception to this would be releases prior to 1.0, which may change as things are rapidly stabilized.

For this issue to be closed we must update the readme with these release details.

Improve UI : List of terminated repair runs

The list of terminated repair runs can grow quite quickly and make the repair screen barely usable.

The form should move above the list of repairs, and we should use an infinite scrolling technique to display past repair jobs.

We would display all unfinished runs + the last 10 terminated repairs at first, then upon scroll we would display older repair runs.

If infinite scroll is hard to achieve, we can make an history screen to display all past repair runs (paginated or not).

discuss setting up mailing list

We will probably want to set up a dedicated mailing list in the near future for Reaper. My personal preference is Google Groups. Thoughts?

fix logging levels and defaults

Logging is a bit insane. We need to go through and make reaper logging useful by decreasing the log levels of certain messages and set up reasonable defaults out of the box.

UI parallelism options don't match Cassandra

The UI stores the string description value of the parallelism option when creating a repair, but when Reaper tries to retrieve the repair from the database, it fails with the following stack trace:

ERROR [2016-12-14 20:13:30,824] [dw-22 - GET /repair_run] i.d.j.e.LoggingExceptionMapper - Error handling a request: 18b1eccefcda9f42 java.lang.IllegalArgumentException: No enum constant org.apache.cassandra.repair.RepairParallelism.parallel at java.lang.Enum.valueOf(Enum.java:238) ~[na:1.8.0_102] at org.apache.cassandra.repair.RepairParallelism.valueOf(RepairParallelism.java:26) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT] at com.spotify.reaper.storage.postgresql.RepairRunMapper.map(RepairRunMapper.java:41) (deleted the rest)

I can (temporarily) get the repair in the database to appear in the UI by changing the value in repair_run.repair_parallelism to the enumeration name (e.g. PARALLEL instead of parallel).

This is reaper 0.3.2 against Cassandra 2.25 using Postgres as a storage backend.

update readme with cassandra storage information

Right now it just says how to use postgres, we need to add C* storage config information

H2 Storage

Currently available as spotify/cassandra-reaper#73, needs to be pulled in. There's conflicts, so a PR with the merged conflicts against master is what we need.

Enable configuration of adress translator

When the issue in dropwizard cassandra component is fixed, the dependency should be updated.

Unable schedule repairs

Hi,

I'm trying to schedule repair on my cluster, but with no luck. I get {"message":"There was an error processing your request. It has been logged (ID af6991042de7f785)."} response in UI and here's the log:

INFO   [2017-03-10 08:18:11,384] [dw-172 - POST /repair_schedule?clusterName=productioncluster&keyspace=accounts&owner=rfurmanski&scheduleTriggerTime=2017-03-10T23%3A59&scheduleDaysBetween=3&repairParallelism=SEQUENTIAL&intensity=0.95&incrementalRepair=false] c.s.r.r.RepairScheduleResource - add repair schedule called with: clusterName = Optional.of(productioncluster), keyspace = Optional.of(accounts), tables = Optional.absent(), owner = Optional.of(rfurmanski), segmentCount = Optional.absent(), repairParallelism = Optional.of(SEQUENTIAL), intensity = Optional.of(0.95), incrementalRepair = Optional.of(false), scheduleDaysBetween = Optional.of(3), scheduleTriggerTime = Optional.of(2017-03-10T23:59) 
INFO   [2017-03-10 08:18:11,431] [dw-172 - POST /repair_schedule?clusterName=productioncluster&keyspace=accounts&owner=rfurmanski&scheduleTriggerTime=2017-03-10T23%3A59&scheduleDaysBetween=3&repairParallelism=SEQUENTIAL&intensity=0.95&incrementalRepair=false] c.s.r.r.RepairScheduleResource - first schedule activation will be: 2017-03-10T23:59:00Z 
ERROR  [2017-03-10 08:18:11,436] [dw-172 - POST /repair_schedule?clusterName=productioncluster&keyspace=accounts&owner=rfurmanski&scheduleTriggerTime=2017-03-10T23%3A59&scheduleDaysBetween=3&repairParallelism=SEQUENTIAL&intensity=0.95&incrementalRepair=false] i.d.j.e.LoggingExceptionMapper - Error handling a request: c42c96a4a6402f4b 
java.lang.NullPointerException: null
	at com.spotify.reaper.resources.RepairScheduleResource.addRepairSchedule(RepairScheduleResource.java:152) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_101]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_101]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at io.dropwizard.jersey.guava.OptionalResourceMethodDispatchAdapter$OptionalRequestDispatcher.dispatch(OptionalResourceMethodDispatchAdapter.java:37) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:295) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at io.dropwizard.jetty.BiDiGzipFilter.doFilter(BiDiGzipFilter.java:127) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:29) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:44) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:248) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:211) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:51) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:92) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:162) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.server.Server.handle(Server.java:446) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]

Can you help me?

Cassandra Storage

currently in PR #2

Exception logging when doing is repair running check

JmxProxy is doing a check if repair is running or not. Between pre 2.2 and post 2.2 version MX bean names have changes and check if repair is running will always issue InstanceNotFoundException in one of the checks which will get logged and pollute logs even though it is expected.

Version of Cassandra should be checked and only one way of checking should be done, or only exception message should be logged since it is expected to have exception here.

Changelog

Hi folks. Thanks for your work on reaper!
To track your changes a simple changelog.md would be awesome.

Network spikes on coordinator (seed node) for repair

During repair run we noticed big network spikes on one node in cluster. This was due to communication between monitoring machine and this single node, to get notifications about ongoing repairs.

This node is added as seed to Reaper through UI. There should be possibility to add multiple seeds when defining cluster so network load for communication can be spread across multiple nodes.

Cassandra 3.0.x + Cassandra Storage bug

I've setup my cassandra-reaper config as follows:

segmentCount: 200
repairParallelism: DATACENTER_AWARE
repairIntensity: 0.9
repairRunThreadCount: 15
hangingRepairTimeoutMins: 30
incrementalRepair: false
scheduleDaysBetween: 7
storageType: cassandra
enableCrossOrigin: true

logging:
  level: DEBUG
  loggers:
    io.dropwizard: INFO
    org.eclipse.jetty: WARN
  appenders:
    - type: console
      logFormat: "%-6level [%d] [%t] %logger{5} - %msg %n"

server:
  type: default
  applicationConnectors:
    - type: http
      port: 8080
      bindHost: 0.0.0.0
  adminConnectors:
    - type: http
      port: 8081
      bindHost: 0.0.0.0

database:
  driverClass: org.postgresql.Driver
  user: postgres
  password: postgres
  url: jdbc:postgresql://127.0.0.1/reaper

cassandra:
  clusterName: "StageCassandra"
  contactPoints: ["<ip>", "<ip>", "<ip>"]
  keyspace: reaper_db

And, after creating the keyspace per https://github.com/thelastpickle/cassandra-reaper/blob/master/src/main/db/reaper_cassandra_db.cql

I get this error after trying to add my first repair.

java.lang.IllegalArgumentException: No enum constant org.apache.cassandra.repair.RepairParallelism.dc_parallel
	at java.lang.Enum.valueOf(Enum.java:238) ~[na:1.8.0_111]
	at org.apache.cassandra.repair.RepairParallelism.valueOf(RepairParallelism.java:26) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.spotify.reaper.storage.CassandraStorage.buildRepairRunFromRow(CassandraStorage.java:703) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.spotify.reaper.storage.CassandraStorage.getRepairRunsAsync(CassandraStorage.java:260) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.spotify.reaper.storage.CassandraStorage.getRepairRunsForCluster(CassandraStorage.java:223) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.spotify.reaper.resources.RepairRunResource.listRepairRuns(RepairRunResource.java:447) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_111]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at io.dropwizard.jersey.guava.OptionalResourceMethodDispatchAdapter$OptionalRequestDispatcher.dispatch(OptionalResourceMethodDispatchAdapter.java:37) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:348) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at io.dropwizard.jetty.BiDiGzipFilter.doFilter(BiDiGzipFilter.java:127) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:29) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:44) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:248) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:211) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:51) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:92) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:162) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.server.Server.handle(Server.java:446) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

With memory storage, I don't get said errors.

Here's the respective versions:

Cassandra:

apt-cache policy cassandra
cassandra:
  Installed: 3.0.9
  Candidate: 3.0.9
  Version table:
 *** 3.0.9 0
        500 http://www.apache.org/dist/cassandra/debian/ 30x/main amd64 Packages
        100 /var/lib/dpkg/status

Oracle Java:

apt-cache policy oracle-java8-installer
oracle-java8-installer:
  Installed: 8u111+8u111arm-1~webupd8~0
  Candidate: 8u111+8u111arm-1~webupd8~0
  Version table:
 *** 8u111+8u111arm-1~webupd8~0 0
        500 http://ppa.launchpad.net/webupd8team/java/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status

Cassandra-Reaper:

apt-cache policy reaper
reaper:
  Installed: 0.3.2-SNAPSHOT
  Candidate: 0.3.2-SNAPSHOT
  Version table:
 *** 0.3.2-SNAPSHOT 0
        100 /var/lib/dpkg/status

The reaper was built from master yesterday using make deb and oracle Java (plus my two PR's).

Debian packages out of date

We've started playing around with your fork of cassandra-reaper over at Reddit and set out to get things all nicely packaged with the included debian definitions. It looks like these are out of date now. Are these going to be maintained long term? If so, are updates in the plans?

When using cassandra storage type, automatically generate requested keyspace if it doesn't exist.

At present, whenever you startup cassandra-reaper using cassandra as the persistent storage, it doesn't create the keyspace specified in the configuration file if it didn't already exist in the cluster.

Ideally, this would be done in CassandraStorage.java before acquiring the session using the keyspace.

I noticed the following line commented out in the the initialize_db.cql file.

-- CREATE KEYSPACE IF NOT EXISTS reaper_db WITH REPLICATION={'class':'SimpleStrategy', 'replication_factor':3};

The problem with this approach is if the user wants to use a different replication factor or other options with the keyspace creation. Unfortunately, the configuration of the keyspace is also specified within the dropwizard-cassandra dependency so it will be difficult to extend the cassandra: section of the yml file as it is based on that dependency.

Some ideas:

Add additional configuration outside of the cassandra: section that are specific to the type of of storage in use. e.g. storageOptions: { type: cassandra, replicationStrategy: SimpleStrategy, replicationFactor: 3 }
Create a dropwizard command that takes additional command line arguments for the the replication strategy and replication factor

NPE in RepairRunner

Line 364 in service/RepairRunner.java swallows a throwable that it likely shouldn't be swallowing. If I change that line to

LOG.error("Executing SegmentRunner failed: " + t.getMessage(), t);

I end up getting this:

INFO [2016-11-11 16:05:52,220] [clustername:1:139] c.s.r.s.SegmentRunner - It is ok to repair segment '139' on repair run with id '1' INFO [2016-11-11 16:05:52,222] [clustername:1:139] c.s.r.c.JmxProxy - Triggering repair of range (-8790692742742779625,-8790661161171969027] for keyspace "keyspacename" on host 10.20.5.87, with repair parallelism DATACENTER_AWARE, in cluster with Cassandra version '2.2.5' (can use DATACENTER_AWARE 'true'), for column families: [customers] ERROR [2016-11-11 16:05:52,226] [clustername:1:139] c.s.r.s.RepairRunner - Executing SegmentRunner failed: null java.lang.NullPointerException: null at java.util.AbstractCollection.addAll(AbstractCollection.java:343) ~[na:1.8.0_91] at org.apache.cassandra.service.StorageService.forceRepairRangeAsync(StorageService.java:3005) ~[cassandra-reaper-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT] at sun.reflect.GeneratedMethodAccessor95.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91] at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) ~[na:1.8.0_91] at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91] at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) ~[na:1.8.0_91] at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) ~[na:1.8.0_91] at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) ~[na:1.8.0_91] at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) ~[na:1.8.0_91] at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) ~[na:1.8.0_91] at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) ~[na:1.8.0_91] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) ~[na:1.8.0_91] at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) ~[na:1.8.0_91] at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468) ~[na:1.8.0_91] at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) ~[na:1.8.0_91] at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309) ~[na:1.8.0_91] at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401) ~[na:1.8.0_91] at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829) ~[na:1.8.0_91] at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91] at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:324) ~[na:1.8.0_91] at sun.rmi.transport.Transport$1.run(Transport.java:200) ~[na:1.8.0_91] at sun.rmi.transport.Transport$1.run(Transport.java:197) ~[na:1.8.0_91] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_91] at sun.rmi.transport.Transport.serviceCall(Transport.java:196) ~[na:1.8.0_91] at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) ~[na:1.8.0_91] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) ~[na:1.8.0_91] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) ~[na:1.8.0_91] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_91] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) ~[na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]

Not real sure where to go from here.

Add ability to repair a specific set of nodes

Reaper should be able to repair a single node or a given set of nodes.
Currently Reaper creates segments for the whole token ring, but should be able to repair only the segments that one node is responsible (or replica) for.

This would allow to run downtime/outage related repairs in Reaper, which is not currently possible.

better feedback in UI for repair failure and history

The current UI is so minimal it is frequently not very helpful.

Stop using deprecated repair methods

The forceRepairRangeAsync and forceRepairAsync methods have been deprecated for a long time now in the Cassandra codebase.

We need to switch to repairAsync(String keyspace, Map<String, String> options) which is the replacement for all of them, in order to have a reliable and simpler way to trigger repair on various Cassandra versions.

This will help us reduce the various checks we make to determine which repair method should be used and remove a few LoC from the JmxProxy.

Auto-publish Docker images to Docker Hub

After spending some time trying to get this to work, I saved what didn't work for me:

     <properties>
+        <!-- Docker related -->
+        <docker-maven-plugin.version>0.4.13</docker-maven-plugin.version>
+        <service.jar>${project.artifactId}-${project.version}-SNAPSHOT.jar</service.jar>
+        <docker.registry>index.docker.io/</docker.registry>
+        <push.image>false</push.image> <!-- set to true to push, normally only on jenkins build on master -->
     </properties>

         <profile>
             <build>
+                    <plugin>
+                        <groupId>com.spotify</groupId>
+                        <artifactId>docker-maven-plugin</artifactId>
+                        <version>${docker-maven-plugin.version}</version>
+                        <configuration>
+                            <imageName>reaperpom</imageName>
+                            <baseImage>java:8-jre</baseImage>
+                            <entryPoint>["java", "-jar",
+                                "/${project.build.finalName}.jar"]
+                            </entryPoint>
+                            <useConfigFile>true</useConfigFile>
+                            <!-- copy the service's jar file from target into the root directory of the image -->
+                            <resources>
+                                <resource>
+                                    <targetPath>/</targetPath>
+                                    <directory>${project.build.directory}
+                                    </directory>
+                                    <include>${project.build.finalName}.jar
+                                    </include>
+                                </resource>
+                            </resources>
+                        </configuration>
+                    </plugin>
                 </plugins>
             </build>
         </profile>

Unable schedule repairs

Hello I am using cassandra-reaper for repairs. I am able to run repair but I am unable to schedule the repairs.
Please see the attachement, I am unable to select Add schedule, the option is greyed out.

I am using the default config from : resource/cassandra-reaper.yaml

Segment count value in repair status not correct for subrange repair

When repair is started with "Incremental repair : false" the value displayed on UI (and the value stored in back end storage) denotes number of nodes in the cluster instead the number defined in configuration with segmentCount property.

It appears that the problem is in CommonTools class at line:

    RepairRun repairRun = storeNewRepairRun(context, cluster, repairUnit, cause, owner, nodes.keySet().size(), repairParallelism, intensity);

where nodes.keySet().size() is passed as a value of int segments parameter. I assume, in case when incrementarRepair is false, that tokenSegments.size() should be passed instead.

If this is, indeed, a bug, I can fix it. I would like to start contributing to the cassandra-reaper.

Also, what is the role of segmentCount property when incremental repairs are run if that number is just number of nodes? Or I am missing something?

add JMX user & pass to cluster configuration

It's confusing that this can't be configured in the UI. Let's add this to the UI and make it available per-cluster.

Let's be sure to encrypt the password in the DB, and the config file can specify a path to the encryption key. Any suggestions on this, @zznate ?

Pluggable observers

Project board link

From IRC:

ha ha, yeah been thinking about it. Still deciding. We might fork it and tweak some of the code to basically to hook custom listeners/callbacks on the success/failure cases. Basically trigger off a slack alert if the repair failed

It would be very cool if we shipped support for pluggable classes that can register as observers to handle things like failure, etc.

┆Issue is synchronized with this Jira Task by Unito
┆Issue Number: K8SSAND-267

Full repair run fails on C* 2.2

There's a bug in Cassandra 2.2 (fixed in 3.0) that prevents Reaper from running full repairs.
A null check is missing on the datacenters collection :

2.2 >> https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/service/StorageService.java#L3040

3.0 >> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L3086-L3089

Reaper repair seems to "hang"

Problem Description:

After starting repair via the GUI, progress remains at 0/x.
Cassandra nodes calculate their respective token ranges, and then nothing happens.
There were no errors in the Reaper or Cassandra logs. Only a message of acknowledgement that a repair had initiated.
Performing stack trace on the running JVM, once can see that the thread spawning the repair process was waiting on a lock that was never being released.
This occurred on all nodes, and prevented any manually initiated repair process from running. A rolling restart of each node was required, after which one could run a nodetool repair successfully.

Cassandra Cluster Details:

Cassandra 2.2.5 running on Windows Server 2008 R2
6 node cluster, split across 2 DCs, with RF = 3:3.

Reaper Details:

Reaper 0.3.3 running on Windows Server 2008 R2, utilising a PostgreSQL database.

Reaper settings:

Parallism: DC-Aware
Repair Intensity: 0.9
Incremental: true

Don't want to swamp you with more details or unnecessary logs, especially as I'd have to sanitize them before sending them out, so please let me know if there is anything else I can provide, and I'll do my best to get it to you.

Notification Email - Enhancement

Add a field to enter an email address to notify when a repair starts/stops/fails. Currently have a crude ruby script which runs via cron and checks to see if a repair is running and to send an email out

continuous reaping

Perhaps a silly idea, but we should discuss having reaper running constantly, throttling itself back based on metrics like load, dropped messages, etc. Then it could scale back on it's own when load is high and take care of business when load is low

Build Docker Images

Create a docker image for cassandra-reaper as part of the build process. Somewhat of a precursor #51 .

problem starting incremental repair when full repair was executed earlier or vice-versa

while creating new run, if the passed value of incremental repair flag is different from the existing value then it should allow to create new repair_unit instead of getting repair_unit based on cluster name/ keyspace /column combination.

so, if the full run was executed for the clustername/keyspace/column was executed earlier, not able to start new incremental run for the same combination. Reverse is also true.

So as per current solution if i delete the repair_unit then due to referential constraints i need to delete repair_segment and repair_run as well which will delete the run history corresponds to the repaid_unit.

start wiki

move most of readme to the wiki

Add jackson JDK8 serializers to properly deserialize dropwizard-cassandra configuration

I was trying to integrate authentication with cassandra and noticed that the version of dropwizard-cassandra being used utilized java.util.Optional rather than Guava's Optional. Because the version of dropwizard being used in the project doesn't automatically add these serializers to the object mapper, we need to explicitly add them during bootstrap.

webUI uses incorrect urls

After building from scratch with mvn clean package -Pbuild-ui and running with (after switching in the yaml file to the memory backend) java -jar target/cassandra-reaper-X.X.X.jar server resource/cassandra-reaper.yaml, I browse to localhost:8080 and the webui doesn't work.

It looks like it is incorrectly looking for resources at the root /deps.js rather than /webui/deps.js

Incremental repair doesn't work

Tried to use incremental repairs with latest reaper snapshot but sstables didn't get marked with "repaired at" timestamp. And repair has finished too quickly (6min), although previous non incremental repair took ~4 hours.

Reaper log:

INFO   [2017-03-31 18:51:20,979] [dw-48 - POST /repair_run?clusterName=test1&keyspace=testks&owner=user&repairParallelism=PARALLEL&incrementalRepair=true] c.s.r.r.RepairRunResource - add repair run called with: clusterName = Optional.of(test1), keyspace = Optional.of(testks), tables = Optional.absent(), owner = Optional.of(user), cause = Optional.absent(), segmentCount = Optional.absent(), repairParallelism = Optional.of(PARALLEL), intensity = Optional.absent(), incrementalRepair = Optional.of(true)
...
INFO   [2017-03-31 18:57:11,194] [test1:1] c.s.r.s.RepairRunner - Next segment to run : 12
INFO   [2017-03-31 18:57:28,743] [test1:1:12] c.s.r.s.SegmentRunner - It is ok to repair segment '12' on repair run with id '1'
INFO   [2017-03-31 18:57:28,745] [test1:1:12] c.s.r.c.JmxProxy - Triggering repair of range (5700662054527062566,5707996255960520728] for keyspace "testks" on host 192.168.0.10, with repair parallelism parallel, in cluster with Cassandra version '3.0.11' (can use DATACENTER_AWARE 'true'), for column families: []
INFO   [2017-03-31 18:57:28,746] [test1:1:12] c.s.r.s.SegmentRunner - Repair for segment 12 started, status wait will timeout in 18000000 millis
INFO   [2017-03-31 18:57:31,790] [test1:1:12] c.s.r.s.SegmentRunner - Repair command 228 on segment 12 returned with state DONE
INFO   [2017-03-31 18:57:41,202] [test1:1] c.s.r.s.RepairRunner - Repair amount done 12
INFO   [2017-03-31 18:57:41,202] [test1:1] c.s.r.s.RepairRunner - Repairs for repair run #1 done

Log from Cassandra node:

DEBUG [AntiEntropyStage:1] 2017-03-31 18:55:32,963 RepairMessageVerbHandler.java:144 - Got anticompaction request AnticompactionRequest{parentRepairSession=9d508810-1643-11e7-b728-5bfc5ef6abd0} org.apache.cassandra.repair.messages.AnticompactionRequest@cc41c373
INFO  [AntiEntropyStage:1] 2017-03-31 18:55:32,963 ActiveRepairService.java:439 - [repair #9d508810-1643-11e7-b728-5bfc5ef6abd0] Not a global repair, will not do anticompaction

for i in *Data.db; do sstablemetadata $i|grep Repa; done

Repaired at: 0
Repaired at: 0
Repaired at: 0
Repaired at: 0
Repaired at: 0
Repaired at: 0
Repaired at: 0
Repaired at: 0

travis testing

Documentation is out of date

In the configuration section you are linking to

http://dropwizard.io/manual/configuration.html

This link returns 404.

Formalize coding standards regarding testability

It's important that we prioritize code quality and ensure our test coverage is excellent. I'd like to formalize our requirements in order to hold ourselves accountable as well as anyone submitting a PR to the project.

Update Documentation for Cassandra Storage with Auth

Update documentation to show configuration for reaper with storage type of cassandra.

HA Reaper

We should be able to set up multiple reaper instances for HA purposes. Ultimately, it would be useful to have the ability to run reaper on every Cassandra node.

Persistent Cassandra storage, username and password

If I want to use Cassandra itself as a persistent storage provider of Cassandra-reaper and I have set the different password of default user ‘cassandra’, how can I specify those into the resource/cassandra-reaper.yaml file. Looks like it only works for default Cassandra username/password (cassandra/cassandra).