Thread blocked when using Materialized View Sort For the Longest Prefix for CSV that contain more than 11 K rows about cqengine HOT 3 CLOSED

Hmm, I am not sure.

Which kind of persistence are you using? On-heap/off-heap/disk?

The only blocking that can occur indefinitely as far as I know, is when using off-heap or disk persistence, when the application fails to call ResultSet.close() when it has finished with the results. Could that be a culprit? If you are using on-heap persistence that should not be a culprit.

from cqengine.

Hi @npgall ,

I see , m working with application on Heap which is base The event loop must never be blocked, if it is then you will exactly that message from above from thrown because the query into the Cqengine took more time than its expected , so our event look report on that as Thread blocked since the response comes back in acceptable time frame.
I tried to work with Data inside the CSV which is around 11K and the response were taking i check it individually around 12 sec with the current implementation.
is there any recommendation base on cqengine documentation with how much data we should work on the CSV ?
is 11K is valid scenario ,if there was benchmarking with such scenario ?
every API has limit if data , does the CQ Engine have such ?
what is the acceptable time that we should get response from the CQEngine when working with Longest Prefix match with the current implementation ?

Regards Fouad Azem

from cqengine.

I see.

I had assumed when you (and that BlockedThreadChecker utility) said the thread is blocked, that you (or it) meant the thread was blocked waiting to acquire some kind of mutex or lock (i.e. indicating there was some kind of deadlock at play).

But based on your last post, it sounds like by "blocked" it means the thread is just busy? So it means your query just has high latency, as opposed to there being some kind of deadlock occurring. Is that the case?

CSV manipulation is not part of CQEngine, so I guess your implementation of attributes might be parsing the CSV?

Generally CQEngine might invoke attributes many times to process a query. It expects the implementation of an attribute to be fast. So if you are doing expensive computations inside attributes (like parsing CSV on-the-fly) that can cause latency.

To reduce latency, consider parsing your CSV into Java objects before you load them into CQEngine, such that your attributes can then just read fields in the objects which will be fast.

If that is not possible, follow the general guidance in the documentation about adding indexes to accelerate your queries.

You could also look at using the "index" merge strategy instead of the default merge strategy.

The "index" merge strategy aims to avoid invoking attributes as much as possible (it assumes invoking them is expensive), and instead it will try to evaluate the query by doing JOINs between indexes under the hood.

Hope that helps. I will close this because I think this is not an issue with CQEngine itself, but rather with CSV parsing. Feel free to reopen if you need more help on the CQEngine side though.

from cqengine.