acoustid / acoustid-index Goto Github PK

View Code? Open in Web Editor NEW

64.0 3.0 23.0 3.85 MB

Minimalistic search engine searching in audio fingerprints from Chromaprint

License: Other

CMake 3.56% Shell 0.03% C++ 93.30% C 2.88% Dockerfile 0.23%

acoustid audio-fingerprinting search

acoustid-index's People

Contributors

Stargazers

Watchers

acoustid-index's Issues

Getting HTTP/0.9 on all requests

Hello, I was trying this out now and got this

$ curl -s -v -X PUT -d "{\"terms\":[100,200,300]}" -H "Content-Type: application/json" 'http://127.0.0.1:5217/main/_doc/1'
*   Trying 127.0.0.1:5217...
* Connected to 127.0.0.1 (127.0.0.1) port 5217 (#0)
> PUT /main/_doc/1 HTTP/1.1
> Host: 127.0.0.1:5217
> User-Agent: curl/7.85.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 23
>
} [23 bytes data]
* Received HTTP/0.9 when not allowed
* Closing connection 0

I'm running it on windows using docker. this is my logs

2022-12-21 19:30:57 Committed revision 0 0
2022-12-21 19:30:57 Simple server listening on "0.0.0.0" port 6080
2022-12-21 19:30:57 HTTP server listening on "0.0.0.0" port 6081
2022-12-21 19:30:57 Prometheus metrics available at "http://0.0.0.0:6081/_metrics"
2022-12-21 19:31:08 Connected to "172.17.0.1:39068"
2022-12-21 19:31:08 Disconnected from "172.17.0.1:39068"
2022-12-21 19:31:09 Connected to "172.17.0.1:39072"
2022-12-21 19:31:09 Disconnected from "172.17.0.1:39072"
2022-12-21 19:32:55 Connected to "172.17.0.1:39074"
2022-12-21 19:32:55 Disconnected from "172.17.0.1:39074"
2022-12-21 19:33:01 Connected to "172.17.0.1:39076"
2022-12-21 19:33:01 Disconnected from "172.17.0.1:39076"
2022-12-21 19:33:08 Connected to "172.17.0.1:39078"
2022-12-21 19:33:08 Disconnected from "172.17.0.1:39078"
2022-12-21 19:33:13 Connected to "172.17.0.1:39080"
2022-12-21 19:33:13 Disconnected from "172.17.0.1:39080"
2022-12-21 19:33:24 Connected to "172.17.0.1:39082"
2022-12-21 19:33:24 Disconnected from "172.17.0.1:39082"
2022-12-21 19:33:38 Connected to "172.17.0.1:39084"
2022-12-21 19:33:38 Disconnected from "172.17.0.1:39084"
2022-12-21 19:33:44 Connected to "172.17.0.1:39086"
2022-12-21 19:33:44 Disconnected from "172.17.0.1:39086"
2022-12-21 19:34:24 Connected to "172.17.0.1:39088"
2022-12-21 19:34:24 Disconnected from "172.17.0.1:39088"
2022-12-21 19:34:44 Connected to "172.17.0.1:39090"
2022-12-21 19:34:44 Disconnected from "172.17.0.1:39090"
2022-12-21 19:34:57 Connected to "172.17.0.1:39092"
2022-12-21 19:34:57 Disconnected from "172.17.0.1:39092"
2022-12-21 19:35:04 Connected to "172.17.0.1:39094"
2022-12-21 19:35:04 Disconnected from "172.17.0.1:39094"

v2 - Segmentation fault when flushing / indexing a large database

I got a segfault indexing a large database (92125 songs).

Segfault also happens when restarting the server to flush the oplogs;
Running fpi-server with valgrind -v returns the following:

.......
Applying oplog entry 49475
Applying oplog entry 49476
Reading oplog entries from 49476
No more oplog entries
Index opened
Creating new staging area 3
Persisting operations up to 0 from staging area 2
Searching in staging area 3
Searching in staging area 3
Searching in staging area 2
Searching in staging area 3
Searching in staging area 3
Searching in staging area 2
Searching in main index
==3913184== Thread 12 Thread (pooled):
==3913184== Invalid read of size 8
==3913184==    at 0x4DD9F4C: QMetaObject::invokeMethodImpl(QObject*, QtPrivate::QSlotObjectBase*, Qt::ConnectionType, void*) (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==  Address 0x1c9d46b8 is 8 bytes inside a block of size 24 free'd
==3913184==    at 0x48399AB: free (vg_replace_malloc.c:538)
==3913184==    by 0x4E562C1: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E57899: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E567AA: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E56F27: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E56CEC: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E56F27: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E57E4F: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E49830: QJsonDocument::fromJson(QByteArray const&, QJsonParseError*) (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x30DE5E: Acoustid::Oplog::read(std::vector<Acoustid::OplogEntry, std::allocator<Acoustid::OplogEntry> >&, int, long) (in /opt/acoustid-index-v2/fpi-server)
==3913184==    by 0x2E4243: Acoustid::Index::open(bool) (in /opt/acoustid-index-v2/fpi-server)
==3913184==    by 0x2E38D9: Acoustid::Index::Index(QSharedPointer<Acoustid::Directory>, bool) (in /opt/acoustid-index-v2/fpi-server)
==3913184==  Block was alloc'd at
==3913184==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==3913184==    by 0x4BFA170: QArrayData::allocate(unsigned long, unsigned long, unsigned long, QFlags<QArrayData::AllocationOption>) (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4C4979E: QByteArray::fromRawData(char const*, int) (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E56334: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E57899: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E567AA: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E56F27: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E56CEC: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E56F27: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E57E4F: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4E49830: QJsonDocument::fromJson(QByteArray const&, QJsonParseError*) (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x30DE5E: Acoustid::Oplog::read(std::vector<Acoustid::OplogEntry, std::allocator<Acoustid::OplogEntry> >&, int, long) (in /opt/acoustid-index-v2/fpi-server)
==3913184==
==3913184== Invalid read of size 8
==3913184==    at 0x4DF9914: QObject::thread() const (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4DD9F50: QMetaObject::invokeMethodImpl(QObject*, QtPrivate::QSlotObjectBase*, Qt::ConnectionType, void*) (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x26DE37: _ZN11QMetaObject12invokeMethodIZZZNK8Acoustid6Server10HttpRouter6handleEPN5qhttp6server12QHttpRequestEPNS5_13QHttpResponseEENKUlvE_clEvENKUlvE_clEvEUlvE_EENSt9enable_ifIXaaaantsrN9QtPrivate15FunctionPointerIT_EE25IsPointerToMemberFunctioneqsrSH_13ArgumentCountLin1EntsrSt14is_convertibleISG_PKcE5valueEbE4typeEP7QObjectSG_N2Qt14ConnectionTypeEPDTclfL0p0_EE (in /opt/acoustid-index-v2/fpi-server)
==3913184==    by 0x26DA4C: Acoustid::Server::HttpRouter::handle(qhttp::server::QHttpRequest*, qhttp::server::QHttpResponse*) const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (in /opt/acoustid-index-v2/fpi-server)
==3913184==    by 0x26E489: QtConcurrent::StoredFunctorCall0<void, Acoustid::Server::HttpRouter::handle(qhttp::server::QHttpRequest*, qhttp::server::QHttpResponse*) const::{lambda()#1}::operator()() const::{lambda()#1}>::runFunctor() (in /opt/acoustid-index-v2/fpi-server)
==3913184==    by 0x26E843: QtConcurrent::RunFunctionTask<void>::run() (in /opt/acoustid-index-v2/fpi-server)
==3913184==    by 0x4BF1ED1: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4BEEBE0: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x592AEA6: start_thread (pthread_create.c:477)
==3913184==    by 0x585ADEE: clone (clone.S:95)
==3913184==  Address 0x38 is not stack'd, malloc'd or (recently) free'd
==3913184==
==3913184==
==3913184== Process terminating with default action of signal 11 (SIGSEGV)
==3913184==  Access not within mapped region at address 0x38
==3913184==    at 0x4DF9914: QObject::thread() const (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4DD9F50: QMetaObject::invokeMethodImpl(QObject*, QtPrivate::QSlotObjectBase*, Qt::ConnectionType, void*) (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x26DE37: _ZN11QMetaObject12invokeMethodIZZZNK8Acoustid6Server10HttpRouter6handleEPN5qhttp6server12QHttpRequestEPNS5_13QHttpResponseEENKUlvE_clEvENKUlvE_clEvEUlvE_EENSt9enable_ifIXaaaantsrN9QtPrivate15FunctionPointerIT_EE25IsPointerToMemberFunctioneqsrSH_13ArgumentCountLin1EntsrSt14is_convertibleISG_PKcE5valueEbE4typeEP7QObjectSG_N2Qt14ConnectionTypeEPDTclfL0p0_EE (in /opt/acoustid-index-v2/fpi-server)
==3913184==    by 0x26DA4C: Acoustid::Server::HttpRouter::handle(qhttp::server::QHttpRequest*, qhttp::server::QHttpResponse*) const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (in /opt/acoustid-index-v2/fpi-server)
==3913184==    by 0x26E489: QtConcurrent::StoredFunctorCall0<void, Acoustid::Server::HttpRouter::handle(qhttp::server::QHttpRequest*, qhttp::server::QHttpResponse*) const::{lambda()#1}::operator()() const::{lambda()#1}>::runFunctor() (in /opt/acoustid-index-v2/fpi-server)
==3913184==    by 0x26E843: QtConcurrent::RunFunctionTask<void>::run() (in /opt/acoustid-index-v2/fpi-server)
==3913184==    by 0x4BF1ED1: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x4BEEBE0: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.2)
==3913184==    by 0x592AEA6: start_thread (pthread_create.c:477)
==3913184==    by 0x585ADEE: clone (clone.S:95)
==3913184==  If you believe this happened as a result of a stack
==3913184==  overflow in your program's main thread (unlikely but
==3913184==  possible), you can try to increase the size of the
==3913184==  main thread stack using the --main-stacksize= flag.
==3913184==  The main thread stack size used in this run was 8388608.
==3913184==
==3913184== HEAP SUMMARY:
==3913184==     in use at exit: 3,545,715,273 bytes in 88,572,691 blocks
==3913184==   total heap usage: 364,892,583 allocs, 276,319,892 frees, 20,566,969,158 bytes allocated
==3913184==
Killed

Make it possible to update fingerprints in acoustid-index

New HTTP API for acoustid-index

Running the index server

Hello.

I'm looking to find songs based on partial clips ( 5-10 seconds ) I have. I can't find a way to do this using the normal API, so my understanding is I have to get a local copy of the database, and do the searches here myself. This tool looks like it'd actually do the searches the way I need, however I do not know:

How do I compile this / get it running, there's no information on that
How do I get a local copy of the database? I've seen a lot of csvs of deltas online, and have downloaded them, but haven't found any information on how to transform these into an usable database.

Thank you very much in advance.
Cheers.

Implement index replication from an external source

met invalid fingerprint while input fpcalc result to index

insert 1 4107786911,4103601822,4105567886,4109760142,4118135454,4151628718,4151636463,4134855151,3069501933,4143244525,4151669997,4151659757,3044396541,3044539741,3043491103,3085425951,3085425950,3086351678,3077961022,3044407103,4118246239,4126634975,4126627229,4126627261,4126504365,4118044141,4118044157,4118048221
ERR invalid fingerprint

change audio fingerprints

Tell me how to change a song sufficiently that the audio fingerprint changes enough to make the song pass filters looking for duplicate songs. I'm looking for advice that works for most stringent filters. Tell me what to do, code, program already available or algorithm. Absolutely anything that works is acceptable.

Http protocol for index/search

Now that acoustid-index has http server for metrics, what if protocol for index INSERT / search would also be available on normal HTTP or GRPC protocol, not only on telnet-like socket.

That would greatly simplify writing client for a new language.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.