Adding optional Xapian support about deft HOT 12 OPEN

This looks interesting! I'd like to check this out, but how can I do that? I don't see the deft-use-xapian option in the xeft.el on your Github. What am I missing?

from deft.

I forgot to add the link to my fork, sorry. Here it is: https://github.com/casouri/deft/tree/xapian Note that the branch is xapian, not master. You can also give Xeft a try to see the potential of Xapian-backend.

To use the fork you need to set deft-xapian-database and set deft-use-xapian to t. I also suggest setting deft-file-limit to like 30 because inserting file summaries for too many results will nullify any speed gained by xapian.

from deft.

Thanks for the addition. I'm having trouble using Xapian on my macOS machine. Installed via brew, but then it asks me for the dynamic module. When I say to compile, it says compilation successful, but then returns progn: Cannot open load file: No such file or directory, xapian-lite

Anyway, this might not the place to discuss this issues. I'll have to investigate later.

EDIT: Seems like the compilation didn't actually compile anything. And downloading the module works, but then later gives an error (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')) – I guess because it's an M1 mac?

from deft.

Yes, the pre-compiled binary are for intel Macs. You need to compile the module locally. After compiling, do you see xapian-lite.so under the same directory as deft.el? Does running make PREFIX=/opt/local under that directory compile anything?

from deft.

No, doesn't seem to work. This is what I get after make PREFIX=/opt/local/

emacs -q -no-site-file -no-init-file -batch -f batch-byte-compile deft.el

In deft--should-use-xapian:
deft.el:1586:11: Warning: defsubst `deft--should-use-xapian' was used before
    it was defined

In end of data:
deft.el:1656:6: Warning: the function `string-join' is not known to be
    defined.

It does create a deft.elc, but no xapian-lite.so

from deft.

Ah, I know why it doesn't work now. I pushed a fix. Could you pull and try again? You can either run make xapian-lite.so PREFIX=/opt/local in project root or call deft and let it compile for you. Both should work now.

from deft.

Thanks for the follow up. After pulling changes I tried both, and both give the same error:

g++ module/xapian-lite.cc -o xapian-lite.so -shared -fPIC -I/opt/local/include -L/opt/local/lib -lxapian
module/xapian-lite.cc:19:10: fatal error: 'xapian.h' file not found
#include <xapian.h>
         ^~~~~~~~~~
1 error generated.
make: *** [xapian-lite.so] Error 1

I double checked, and brew says xapian is installed.

I tried the xeft repo as well, and it gives the same error after make

from deft.

I thought homebrew on ARM Mac uses /opt/local, turns out it's actually /opt/homebrew. So you want to use make xapian-lite.so PREFIX= /opt/homebrew.

from deft.

That was it! It did compile now, and I'm trying it out. Seems fast indeed!

Could it be that some functionality of Deft is changed because of the Xapian search engine?

I wrote Zetteldeft (https://github.com/efls/zetteldeft), which implements plain text links via the Deft search functions. Following a link consists of searching for a unique ID in the filename. To achieve this, some of the search functions set deft-filter-only-filenames to t. But with Xapian enabled, this setting doesn't seem to be respected, and the Deft search results include both files with the search string in filename as well as file contents.

I haven't looked at the internals of your branch yet, but can you guess why this happens?

from deft.

Yeah, I didn't add support for that variable. Xapian works by indexing words in a file to a database and querying the database when searching, so it can't search regex nor filename. I think it's easier and more efficient to write some elisp to search for filenames (saving all filenames to a buffer and search in the buffer) than implementing this in xapian.

from deft.

Makes sense. Might be important to point out functional differences because of the Xapian engine.

I think it's easier and more efficient to write some elisp to search for filenames (saving all filenames to a buffer and search in the buffer) than implementing this in xapian.

Yeah at some point in the future Zetteldeft should use a non-Deft function to search filenames. Perhaps creating a hash with file paths and filenames IDs or something.

from deft.

Yeah maybe it isn't that good of an idea to use xapian for deft, since there are quite a few deft features that xapian cannot support: regex, filename-only, plus it's non-trivial to ditch the caching.

from deft.