slivne / try_git Goto Github PK

    Check that we can recover from a failure durring restore
    1. Use a single node and create a keyspace + table
    2. Insert data
    3. Create snapshot and save files
    4. Drop keyspace
    5. Create keyspace + table + populate different data
    6. Start restore data
    7. Kill node
    8. Start node
    9. Check if non restored data exists
    10. Check if any restored data exists
    11. Restore data
    12. Check that all data exists

    Check that repair is accompileshed while node is decomissioned
    1. Create a cluster of 3 nodes with rf=2
    2. Stop node 2
    3. Insert data
    4. Start node 2
    6. Start repair
    7. Decomission node 3
    8. Stop node 1 - does node 2 hold all the data

    Check that repair is accompileshed while new node is added
    1. Create a cluster of 2 nodes with rf=2
    2. Stop node 2
    3. Insert data
    4. Start node 2
    6. Start repair
    7. Create a new node and start it

    Check that killing a repairing node does not cause additional failures
    1. Create a cluster of 2 nodes with rf=2
    2. Stop node 2
    3. Insert data
    4. Start node 2
    5. In a loop
       a. Start node 1 if its down
       b. Start repair
       c. Kill node 1
       d. Check that cluster is avilable (read/writes)

    Check that repair fixes a few update to cells contents
    1. Create a cluster of 2 nodes with rf=2, disable read_repair, hinttef_handoff
    2. Insert data
    3. Shutdown node 2
    4. Update some cells
    5. Start node 2
    6. Run repair on node 2
    7. Shutdown node 1 - check that all data exists on node 2

    Check that repair fixes a deletion of cell range
    1. Create a cluster of 2 nodes with rf=2, disable read_repair, hinttef_handoff
    2. Insert data
    3. Shutdown node 2
    4. Delete a range of some cells
    5. Start node 2
    6. Run repair on node 2
    7. Shutdown node 1 - check that all data exists on node 2

    Check that we can restore snapshot files that use old schema
    1. Use a single node and create a keyspace + table
    2. Insert data
    3. Create snapshot and save files
    4. Drop keyspace
    5. Create keyspace + table
    6. Alter table
    7. Restore data
    8. Check that all data exists

    Check that repair fixes all inconsistencies in data
    1. Create a cluster of 3 nodes with rf=3, disable read_repair, hintted_handoff
    2. Shutdown node 2,3
    3. Insert data set A
    4. Start node 2 Shutdown node 1
    5. Insert data set B (A != B)
    6. Start node 3 Shutdown node 2
    7. Insert data set C (A != B, B != C, A != C)
    8. Start node 1,2
    9. Run repair on node 3
    10. Shutdown node 1,2 check all data on node 3
    11. Shutdown node 1,3 check all data on node 2
    12. Shutdown node 2,3 check all data on node 1

    Check that repair is able to complete if one replica of data exists
    1. Create a cluster of 4 nodes with rf=3
    2. Stop node 2
    3. Insert data
    4. Stop node 3
    5. Start node 2
    6. Start repair on node 2
    7. Check if repair is succesfull

    Check that repair transfers all the data in case non exists
    1. Create a cluster of 2 nodes with rf=2, disable read_repair, hintted_handoff
    2. Shutdown node 2 (prior to schema creation)
    3. Insert data
    4. Start node 2
    5. Run repair on node 2
    6. Shutdown node 1 - check that all data exists on node 2

    Check that repair fixes a few removed keys
    1. Create a cluster of 2 nodes with rf=2, disable read_repair, hinttef_handoff
    2. Insert data
    3. Shutdown node 2
    4. Remove some keys
    5. Start node 2
    6. Run repair on node 2
    7. Shutdown node 1 - check that all data exists on node 2

    Check that repair is not able to complete if no replicas for data exist
    1. Create a cluster of 3 nodes with rf=2
    2. Stop node 2
    3. Insert data
    4. Stop node 3
    5. Start node 2
    6. Start repair on node 2
    7. Check if repair is succesfull

    Check that repair fixes a deleteion of cells
    1. Create a cluster of 2 nodes with rf=2, disable read_repair, hinttef_handoff
    2. Insert data
    3. Shutdown node 2
    4. Delete some cells
    5. Start node 2
    6. Run repair on node 2
    7. Shutdown node 1 - check that all data exists on node 2

    Check that incremetal backup works as expected
    1. Use a single node
    2. Enable incremental_backup
    3. Create a keyspace + table
    4. Insert data
    5. Check that while sstables are flushed - incremental backups are created
    6. Run compact -forcing all ssstables to be merged
    7. Check that backups holds all the old files and the new compacted file

    Check that we can restore snapshot files that used a different smp setting
    1. Use a single node with smp=1 and create a keyspace + table
    2. Insert data
    3. Create snapshot and save files
    4. Drop keyspace
    5. Stop node, start it with smp=2
    6. Create keyspace + table
    7. Restore data
    8. Check that all data exists

    Check that repair transfers all the data in case non exists
    1. Create a cluster of 2 nodes with rf=2, disable read_repair, hintted_handoff
    2. Shutdown node 2 (prior to schema creation)
    3. Insert data
    4. Start node 2
    5. Run repair on node 1
    6. Shutdown node 1 - check that all data exists on node 2

    Check that data can be updated while repair is running
    1. Create a cluster of 2 nodes with rf=2
    2. Stop node 2
    3. Insert data
    4. Start node 2
    5. In a loop update part of data with CL=2
    6. Start repair
    7. Stop node 1
    8. Check that all the data is up to date

    Check that killing a repaired node does not cause additional failures
    1. Create a cluster of 2 nodes with rf=2
    2. Stop node 2
    3. Insert data
    4. In a loop
       a. Start node 2
       b. Start repair
       c. Kill node 2
       d. Check that cluster is avilable (read/writes)

    Check that repair fixes updates to ttl
    CQL: UPDATE table USING TTL <ttl value> where key=X
    1. Create a cluster of 2 nodes with rf=2, disable read_repair, hinttef_handoff
    2. Insert data
    3. Shutdown node 2
    4. Update ttl of some cells
    5. Start node 2
    6. Run repair on node 2
    7. Shutdown node 1 - check that all data exists on node 2

    Check that we can restore snapshot files that use old schema
    1. Use a single node and create a keyspace + table
    2. Insert data
    3. Create snapshot and save files
    4. Drop keyspace
    5. Create keyspace + table
    6. Restore data
    7. Check that all data exists
    8. Restore data
    9. Check that all data exists

    Check that we can restore snapshot files that have been created by cassandra
    1. Use a single node and create a keyspace + table
    2. Restore data from a cassandra snapshot
    3. Check that all data exists

    Check that we can restore snapshot files that use a non updated token ownership
    1. Use a single node and create a keyspace + table
    2. Insert data
    3. Create snapshot and save files
    4. Add an additional node
    5. Drop keyspace
    6. Create keyspace + table
    7. Restore data
    8. Check that all data exists

    Check that repair is accompilshed when multiple repairs are initiated in parallel
    1. Create a cluster of 3 nodes with rf=3
    2. Insert data
    3. Stop node 2
    4. Insert data
    5. Stop node 3
    6. Insert data
    7. Start node 2, Start node 3
    8. Start repair on node 2, node 3
    9. Stop node 1,node 3 - does node 2 hold all the data
    10. Stop node 1,node 2 - does node 3 hold all the data

    Check that we can recover from a failure durring snapshot
    1. Use a single node and create a keyspace + table
    2. Insert data
    3. Start create snapshot
    4. Kill node
    5. Start node
    6. Check that all data exists