How exactly do you handle duplicate update? about readablewebproxy HOT 5 CLOSED

At the moment, row versioning is done via https://github.com/kvesteri/sqlalchemy-continuum, which you basically plug in and suddenly you have a changelog on a per-row basis.

Deduplication isn't /really/ done, though I probably need to look into that eventually.

from readablewebproxy.

Thanks!

from readablewebproxy.

NP.

FWIW, right now the indices are the great majority of the db size, so improving indice efficiency and just dropping the indices that aren't queried against from the history table has been a major improvement. Also, the postgresql TOAST and automatic compression do a crazy good job of storing lots of ASCII efficiently.

At one point, I had t GiST indice across the entire text content of every row (I was experimenting with full text search across the entire page), and the index alone was 4-5 times larger then the database content itself.

from readablewebproxy.

Did you ever experimented with using Elastic Search or some other specialized search engine or are you happy with postgresql ("good enough")? Thanks again for your attention!

from readablewebproxy.

I did spend some time looking at different search engines, but at the moment, I'm just not using search at all.

Mostly, it wasn't too useful, and the space costs were somewhat exorbitant. I can do offline queries into the DB if I really want something. It's slow as hell (as it's unindexed), but I can dump the output to a csv for post-analysis if needed.

Yet another thing I'd like to do at some point, and just haven't had time recently.

from readablewebproxy.