Comments (11)
As you probably know, DSE is not an open source project, and its license doesn't allow to use it to improve this project. However, if you can give us information about your use case, environments, query patterns, etc., we will do our best to help you with performance, and any feedback to improve the project will be more than welcome.
from cassandra-lucene-index.
I am doing a simple test on a table with two indexes - one int and one date. I got 3 c* nodes and I've been switching the nodes between DSE+Search and plain cassandra + lucene. The same query expressed with DSE Solr seem to sustain over 5000 concurrent requests within 2 seconds of response time. Lucene index based search is struggling to get over 50 concurrent requests. Just to be sure, I tried to make staggered requests where each request is 7ms apart and your lucene based queries are responding within 15ms average. But with other concurrent reads or writes, the query performance struggles. This suggests there may be a concurrency management issue. If only the reads and writes can be throttled during access in some way, perhaps the overall performance can be improved? A profiling tool to test the concurrency issues may reveal something insightful as well.
from cassandra-lucene-index.
One more thing I noticed - the latency issue I was experiencing apparently does not apply on a single node cassandra. With a single node, the searches on lucene indexes are as fast as the DSE-Solr counterpart. Once I move to a multi-node setup, then the latency dramatically increases.
from cassandra-lucene-index.
As a workaround, I've applied a semaphore block on IndexQueryHandler around the process method to limit concurrency and this at least does not crash outright. I am now able to get over 1000 concurrent requests with 7.5s average latency. I can live with this for my use-case till the concurrency issue is resolved.
from cassandra-lucene-index.
First of all, thanks for your feedback.
Could you give us more information about your tests? Which is your replication factor and consistency level? Are you using vnodes? Are your using queries and/or filters? Are you using rotational or SSD disks? How dependent from caching are your tests? Please feel free to send us whatever you deem relevant to andres at stratio dot com.
Could you please give us more info about your changes in the query handler? There is a concurrencyFactor
variable at LuceneStorageProxy
that could be modified for this. Maybe this concurrency factor could be specified in a per-query basis.
from cassandra-lucene-index.
My tests has the following configs:
Replication factor: 2
Nodes: 3
Disk: Local SSD
No VNodes
Only filters are used - no queries
No Cache - each query in the tests produces unique parameters
Each node has:
- Baremetal 40 core Xeon processors
- 200GB RAM
For the single node test, I used a much less powerful VM but the queries were very fast even on high concurrency. This seems to indicate a network based congestion. My current workaround involves gating the calls to LuceneStorageProxy.getRangeSlice effecitvely throttling it to avoid tripping over with the concurrency issue.
from cassandra-lucene-index.
Ok, thanks. Which is the read consistency level? How many columns have the rows? Are you using a primary key composed by partition and clustering key, or only by partition key? How many rows must be returned by each query? Are you using a limit
clause to reduce the amount of collected rows?
from cassandra-lucene-index.
Read consistency: one
Table has 34 columns
7 of them have lucene indexes (3 long, 1 integer, 1 UUID, 1 date, 1 string)
2 columns are in partition key and 2 more in clustering key. The clustering key is timestamp based.
Each query returns between 1 to 5 rows. The query is limited to max of 100 rows.
from cassandra-lucene-index.
Any updates/new findings on this issue?
from cassandra-lucene-index.
benchmark DSE vs Stratio would be nice (-:
from cassandra-lucene-index.
👍
from cassandra-lucene-index.
Related Issues (20)
- Altering UDTs cause ArrayIndexOutOfBoundsException in ColumnsMapper HOT 14
- Unable to find custom indexer in Cassandra 3.11.2 HOT 1
- Support cassandra-3.11.3
- When a column from the index is used in a predicate cassandra always returns 0 records.
- ArrayIndexOutOfBounds with collections of UDTs HOT 2
- Request timeout error while retrieving data using Stratio-cassandra-lucene-index
- Can't retrieve Many records and count(*)
- Indexing Status
- Index text field as JSON HOT 2
- cassandra search - Returning the score of each result HOT 1
- Query maps where value is case_sensitive=false
- Support for Cassandra 4
- Custom Query
- Exception encountered during startup
- Upgrade to JTS 1.16.1
- Lucene Index Empty after upgrade
- Lucene Index with Cassandra SSL
- Another CQL "date" type: No results
- JVM segfault in ~StubRoutines::updateBytesCRC32 during commit() HOT 1
- Cassandra secondary lucene index rebuild is very slowly
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cassandra-lucene-index.