Giter VIP home page Giter VIP logo

maxime2 / dataparksearch Goto Github PK

View Code? Open in Web Editor NEW
94.0 94.0 45.0 6.32 MB

An open source, feature rich search engine.

Home Page: http://www.dataparksearch.org/

License: GNU General Public License v2.0

Shell 6.03% Makefile 7.73% Slash 1.28% C 65.18% Perl 0.73% XS 0.36% Scilab 0.15% HTML 1.90% Batchfile 0.40% xBase 12.13% M4 1.44% PLpgSQL 0.09% Raku 0.01% POV-Ray SDL 2.46% NASL 0.10% ReScript 0.01%

dataparksearch's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dataparksearch's Issues

search.cgi outputs itself on the screen

I downloaded a latest version master.zip package and installed it in a Linux CentOS 6.4 box.
backed with MySQL 5.5
I run indexer -W and indexed 10000+ pages on my site.
Since I use Zend Framework to program the site and I use directive instead of .htaccess. So I use +ExecCGI to enable the search.cgi to run.
Also, I modified httpd.conf accordingly as the following:

ScriptAlias /cgi-bin/ "mysiteRoot/cgi-bin/"

<Directory "mysiteRoot/cgi-bin/">

After these procedures, I copied /usr/local/dpsearch/bin/search.cgi and /usr/local/dpsearch/etc/search.htm to "mysiteRoot/cgi-bin" and chown and chmod them.

Then I http://www.ypgogo.com/cgi-bin/search.cgi

I can see major part of search.cgi itself was thrown onto the screen with so many errors. Such can't load template.

Is there any instruction that I can write a search to query the index? Please advise. Thanks your efforts!

IPv6 URL support

I am using this engine to index modarate-size IPv6 only network(using squid proxy) where some services don't have domain names and instead use raw IPv6 URLs which look like http://[2001:db8:85a3:8d3:1319:8a2e:370:7348]/. Problem is that Dataparksearch seems to cut the url after first : character. This issue is only about IPv6 URL support, not full IPv6 support.

Here's how the URLs look in database:
Database view

Parameterized Queries

After analyzing the queries ran on the database, it seems as though the lack of parameterized queries affect the efficiency of the sql statements. Are there any plans to update the queries in the sql.c file to use parameterized queries?

gcc5 make failed

gcc version 5.3.0 (FreeBSD Ports Collection):

libtool: link: gcc5 -DHAVE_PTHREAD -pthread -DDPS_CONF_DIR="/usr/local/dpsearch/etc" -DDPS_VAR_DIR="/usr/local/dpsearch/var" -DDPS_SHARE_DIR="/usr/local/dpsearch/share" -g -O3 -march=native -mtune=native -I/usr/local/include -I/usr/local/include -o .libs/filler.cgi filler_cgi-filler.o -pthread -pthread ./.libs/libdpsearch.so /usr/home/aiia/dpsearch-git/trunk/src/.libs/libdp.so -L/usr/local/lib -L/usr/local/lib/mysql ./.libs/libdpcharset.so -lpthread ./.libs/libdp.so -lcares -lidn -liconv -lperconaserverclient -lz -lm -pthread -Wl,-rpath -Wl,/usr/local/dpsearch/lib
gcc5 -DHAVE_CONFIG_H -I. -I../include -I./../include -I/usr/local/include -I/usr/local/include -I/usr/local/include/mysql -DHAVE_PTHREAD -pthread -DDPS_CONF_DIR="/usr/local/dpsearch/etc" -DDPS_VAR_DIR="/usr/local/dpsearch/var" -DDPS_SHARE_DIR="/usr/local/dpsearch/share" -g -O3 -march=native -mtune=native -I/usr/local/include -I/usr/local/include -MT indexer-main.o -MD -MP -MF .deps/indexer-main.Tpo -c -o indexer-main.o test -f 'main.c' || echo './'main.c
main.c: In function 'main':
main.c:1380:6: error: '_malloc_options' undeclared (first use in this function)
_malloc_options = "ax3N";
^
main.c:1380:6: note: each undeclared identifier is reported only once for each function it appears in
*** Error code 1

Stop.
make[2]: stopped in /usr/home/aiia/dpsearch-git/trunk/src
*** Error code 1

Stop.
make[1]: stopped in /usr/home/aiia/dpsearch-git/trunk/src
*** Error code 1

Stop.
make: stopped in /usr/home/aiia/dpsearch-git/trunk

acronym.c:269:sequence point error ?

acronym.c:269:117: warning: operation on '(ww + (sizetype)(i * 48))->len' may be undefined [-Wsequence-point]

Source code is

 DpsConv(&uni_lc, ww[i].word, (15*ww[i].len+1)*sizeof(char), (char*)ww[i].uword, sizeof(dpsunicode_t)*((ww[i].len = DpsUniLen(ww[i].uword)) + 1));

undefined reference to 'DpsStrndup'

Hi Maxime,

I'm trying to build the project on Debian and I ran into the following issue:

/bin/bash ../libtool --tag=CC --mode=link gcc -DHAVE_PTHREAD -pthread -DDPS_CONF_DIR=\"/usr/local/dpsearch/etc\" -DDPS_VAR_DIR=\"/usr/local/dpsearch/var\" -DDPS_SHARE_DIR=\"/usr/local/dpsearch/share\" -g -O2 -O2 -mcmodel=medium -o search.cgi search_cgi-search.o libdpsearch.la libdpcharset.la -pthread -lpthread libdp.la -L/usr/local/lib -lrt -lc -lresolv -lm -lcares -lz -lreadline -ltermcap -lidn -L/usr/lib/x86_64-linux-gnu -lmysqlclient -lpthread -lz -lm -ldl -ldl -L/usr/lib/x86_64-linux-gnu -lpq -lcrypt -lssl -lcrypto -L/usr/local/lib -lgc libtool: link: gcc -DHAVE_PTHREAD -pthread -DDPS_CONF_DIR=\"/usr/local/dpsearch/etc\" -DDPS_VAR_DIR=\"/usr/local/dpsearch/var\" -DDPS_SHARE_DIR=\"/usr/local/dpsearch/share\" -g -O2 -O2 -mcmodel=medium -o .libs/search.cgi search_cgi-search.o -pthread ./.libs/libdpsearch.so ./.libs/libdpcharset.so ./.libs/libdp.so -L/usr/local/lib -lrt -lc -lresolv -lcares -lreadline -ltermcap -lidn -L/usr/lib/x86_64-linux-gnu -lmysqlclient -lpthread -lz -lm -ldl -lpq -lcrypt -lssl -lcrypto -lgc -pthread -Wl,-rpath -Wl,/usr/local/dpsearch/lib ./.libs/libdpsearch.so: undefined reference to 'DpsStrndup' collect2: error: ld returned 1 exit status Makefile:1093: recipe for target 'search.cgi' failed make[2]: *** [search.cgi] Error 1

When configuring the project with: ./configure --enable-pophops --enable-reltrack --enable-boehmgc --enable-php-module --enable-idn --with-optimise --with-openssl --with-readline --with-aspell --with-extra-charsets=all --with-mysql --with-pgsql

Any hints on what might be wrong?

Thanks!

Attempt to free invalid pointer

compiled with: ./configure --enable-googlegrp --enable-pophops --enable-reltrack --enable-boehmgc --enable-idn --with-debug --with-optimise --with-openssl --with-aspell --with-readline --with-mysql

ran with: ./sbin/indexer -o -r -b -q -U -v 5

output:

GC Warning: Repeated allocation of very large block (appr. size 5578752):
	May lead to memory leak and poor performance.
Wed 07 21:48:15 [28969] Can't create '/usr/local/dpsearch/var/indexer.pid' - (17) File exists
Wed 07 21:48:15 [28969] Process 28883
 seems to be dead. Flushing '/usr/local/dpsearch/var/indexer.pid' - (3) No such process
indexer[28969]: {00} DpsOpenCache:
indexer[28969]: {00} i:0  cached_sd:0  sin_port:0
GC Warning: Repeated allocation of very large block (appr. size 144732160):
	May lead to memory leak and poor performance.
indexer[28969]: {00} wrd_buf: 127ce020
indexer[28969]: {00} Done.
indexer[28969]: {00} indexer from dpsearch-4.54-2017-09-05-mysql started with '/usr/local/dpsearch/etc/indexer.conf'
indexer[28969]: {00} Chinese dictionary with 0 entries
indexer[28969]: {00} Korean dictionary with 0 entries
indexer[28969]: {00} Thai dictionary with 0 entries
indexer[28969]: {00} LogsOnly: no
indexer[28969]: {00} mutexes used: 4105
indexer[28969]: {00} The following sections are defined
indexer[28969]: {00} Sections.body [1,256:00]: <NULL>
indexer[28969]: {00} Sections.Charset [12,32:00]: <NULL>
indexer[28969]: {00} Sections.Content-Language [14,16:00]: <NULL>
indexer[28969]: {00} Sections.Content-Type [13,64:00]: <NULL>
indexer[28969]: {00} Sections.crosswords [11,0:00]: <NULL>
indexer[28969]: {00} Sections.meta.description [4,128:00]: <NULL>
indexer[28969]: {00} Sections.meta.keywords [3,128:00]: <NULL>
indexer[28969]: {00} Sections.title [2,128:00]: <NULL>
indexer[28969]: {00} Sections.url [6,0:01]: <NULL>
indexer[28969]: {00} URL: http://www.tria-alfa.ro
indexer[28969]: {00} Server Allow 'www.surse.ro'
indexer[28969]: {00} Allow by default
indexer[28969]: {00} IDN: www.tria-alfa.ro [ISO-8859-1] -> www.tria-alfa.ro
Attempt to free invalid pointer 0x1b71ba0
Invalid pointer passed to free()

indexer fails due to "Floating point exception (core dumped)"

gdb backtrace:

(gdb) set args -W
(gdb) run
Starting program: /usr/local/dpsearch/sbin/indexer -W
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGFPE, Arithmetic exception.
0x00007ffff7985efb in DpsURLAction (A=A@entry=0x638160 <Main>, D=D@entry=0x7fffffff8090, cmd=cmd@entry=2) at db.c:858
858           dbfrom = dbto = D->id % A->Conf->dbl.nitems;
(gdb) bt
#0  0x00007ffff7985efb in DpsURLAction (A=A@entry=0x638160 <Main>, D=D@entry=0x7fffffff8090, cmd=cmd@entry=2) at db.c:858
#1  0x00007ffff791a70e in DpsStoreHrefs (Indexer=Indexer@entry=0x638160 <Main>) at indexer.c:444
#2  0x00007ffff795333f in DpsEnvLoad (Indexer=Indexer@entry=0x638160 <Main>, cname=cname@entry=0x644660 <cname> "/usr/local/dpsearch/etc/indexer.conf", lflags=<optimized out>)
    at conf.c:2387
#3  0x00007ffff7920bb7 in DpsIndexerEnvLoad (Indexer=Indexer@entry=0x638160 <Main>, fname=fname@entry=0x644660 <cname> "/usr/local/dpsearch/etc/indexer.conf", lflags=<optimized out>)
    at indexer.c:2896
#4  0x000000000040307c in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at main.c:1503
(gdb)

I put

Server http://www.spiegel.de/

into /usr/local/dpsearch/etc/indexer.conf.

experienced with 9257ced

sudo make install fails due to overwriting /usr/local/dpsearch/include/dps_http.h

After successful make && DPS_TEST_DBADDR='localhost:5432/datapark_test' make check I face

$ LANGUAGE=en sudo make install
Making install in include
make[1]: Entering directory `/mnt/sources_image_btrfs/dpsearch-4.53/include'
make[2]: Entering directory `/mnt/sources_image_btrfs/dpsearch-4.53/include'
make[2]: Nothing to be done for `install-exec-am'.
 /bin/mkdir -p '/usr/local/dpsearch/include'
 /usr/bin/install -c -m 644 dps_common.h dps_conf.h dps_config.h dps_db.h dps_db_int.h dps_hrefs.h dps_indexer.h dps_log.h dps_url.h dps_spell.h dps_mutex.h dps_utils.h dps_charsetutils.h dpsearch.h dps_search_tl.h dps_ftp.h dps_socket.h dps_host.h dps_xmalloc.h dps_id3.h dps_http.h dps_http.h dps_cache.h dps_boolean.h dps_server.h dps_searchtool.h dps_searchcache.h dps_alias.h dps_env.h dps_agent.h dps_word.h dps_parsehtml.h dps_stopwords.h dps_parser.h dps_proto.h dps_robots.h dps_signals.h dps_execget.h dps_services.h dps_doc.h '/usr/local/dpsearch/include'
/usr/bin/install: will not overwrite just-created ‘/usr/local/dpsearch/include/dps_http.h’ with ‘dps_http.h’
make[2]: *** [install-includeHEADERS] Error 1
make[2]: Leaving directory `/mnt/sources_image_btrfs/dpsearch-4.53/include'
make[1]: *** [install-am] Error 2
make[1]: Leaving directory `/mnt/sources_image_btrfs/dpsearch-4.53/include'
make: *** [install-recursive] Error 1

might be related to

$ DPS_TEST_DBADDR='localhost:5432/datapark_test' make distcheck 
make  dist-bzip2 am__post_remove_distdir='@:'
make[1]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53'
if test -d "dpsearch-4.53"; then find "dpsearch-4.53" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "dpsearch-4.53" || { sleep 5 && rm -rf "dpsearch-4.53"; }; else :; fi
test -d "dpsearch-4.53" || mkdir "dpsearch-4.53"
 (cd include && make  top_distdir=../dpsearch-4.53 distdir=../dpsearch-4.53/include \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[2]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/include'
make[2]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/include'
 (cd src && make  top_distdir=../dpsearch-4.53 distdir=../dpsearch-4.53/src \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[2]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/src'
make[2]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/src'
 (cd doc && make  top_distdir=../dpsearch-4.53 distdir=../dpsearch-4.53/doc \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[2]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/doc'
make  \
      top_distdir="../dpsearch-4.53" distdir="../dpsearch-4.53/doc" \
      dist-hook
make[3]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/doc'
cp -rf ./samples ../dpsearch-4.53/doc
make[3]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/doc'
make[2]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/doc'
 (cd etc && make  top_distdir=../dpsearch-4.53 distdir=../dpsearch-4.53/etc \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[2]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/etc'
make[2]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/etc'
 (cd create && make  top_distdir=../dpsearch-4.53 distdir=../dpsearch-4.53/create \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[2]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create'
 (cd ibase && make  top_distdir=../../dpsearch-4.53 distdir=../../dpsearch-4.53/create/ibase \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[3]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/ibase'
make[3]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/ibase'
 (cd mssql && make  top_distdir=../../dpsearch-4.53 distdir=../../dpsearch-4.53/create/mssql \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[3]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/mssql'
make[3]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/mssql'
 (cd mysql && make  top_distdir=../../dpsearch-4.53 distdir=../../dpsearch-4.53/create/mysql \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[3]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/mysql'
make[3]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/mysql'
 (cd oracle && make  top_distdir=../../dpsearch-4.53 distdir=../../dpsearch-4.53/create/oracle \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[3]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/oracle'
make[3]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/oracle'
 (cd pgsql && make  top_distdir=../../dpsearch-4.53 distdir=../../dpsearch-4.53/create/pgsql \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[3]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/pgsql'
make[3]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/pgsql'
 (cd sqlite && make  top_distdir=../../dpsearch-4.53 distdir=../../dpsearch-4.53/create/sqlite \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[3]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/sqlite'
make[3]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/sqlite'
 (cd mimer && make  top_distdir=../../dpsearch-4.53 distdir=../../dpsearch-4.53/create/mimer \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[3]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/mimer'
make[3]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create/mimer'
make[2]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/create'
 (cd misc && make  top_distdir=../dpsearch-4.53 distdir=../dpsearch-4.53/misc \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[2]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/misc'
make[2]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/misc'
 (cd scripts && make  top_distdir=../dpsearch-4.53 distdir=../dpsearch-4.53/scripts \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[2]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/scripts'
make[2]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/scripts'
 (cd test && make  top_distdir=../dpsearch-4.53 distdir=../dpsearch-4.53/test \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[2]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/test'
make[2]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/test'
make  \
      top_distdir="dpsearch-4.53" distdir="dpsearch-4.53" \
      dist-hook
make[2]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53'
cp -rf ./debian dpsearch-4.53
make[2]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53'
test -n "" \
    || find "dpsearch-4.53" -type d ! -perm -755 \
        -exec chmod u+rwx,go+rx {} \; -o \
      ! -type d ! -perm -444 -links 1 -exec chmod a+r {} \; -o \
      ! -type d ! -perm -400 -exec chmod a+r {} \; -o \
      ! -type d ! -perm -444 -exec /bin/bash /home/richter/sources/dpsearch-4.53/install-sh -c -m a+r {} {} \; \
    || chmod -R a+r "dpsearch-4.53"
tardir=dpsearch-4.53 && ${TAR-tar} chof - "$tardir" | BZIP2=${BZIP2--9} bzip2 -c >dpsearch-4.53.tar.bz2
make[1]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53'
if test -d "dpsearch-4.53"; then find "dpsearch-4.53" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "dpsearch-4.53" || { sleep 5 && rm -rf "dpsearch-4.53"; }; else :; fi
case 'dpsearch-4.53.tar.bz2' in \
    *.tar.gz*) \
      GZIP=--best gzip -dc dpsearch-4.53.tar.gz | ${TAR-tar} xf - ;;\
    *.tar.bz2*) \
      bzip2 -dc dpsearch-4.53.tar.bz2 | ${TAR-tar} xf - ;;\
    *.tar.lz*) \
      lzip -dc dpsearch-4.53.tar.lz | ${TAR-tar} xf - ;;\
    *.tar.xz*) \
      xz -dc dpsearch-4.53.tar.xz | ${TAR-tar} xf - ;;\
    *.tar.Z*) \
      uncompress -c dpsearch-4.53.tar.Z | ${TAR-tar} xf - ;;\
    *.shar.gz*) \
      GZIP=--best gzip -dc dpsearch-4.53.shar.gz | unshar ;;\
    *.zip*) \
      unzip dpsearch-4.53.zip ;;\
    esac
chmod -R a-w dpsearch-4.53
chmod u+w dpsearch-4.53
mkdir dpsearch-4.53/_build dpsearch-4.53/_inst
chmod a-w dpsearch-4.53
test -d dpsearch-4.53/_build || exit 0; \
    dc_install_base=`CDPATH="${ZSH_VERSION+.}:" && cd dpsearch-4.53/_inst && pwd | sed -e 's,^[^:\\/]:[\\/],/,'` \
      && dc_destdir="${TMPDIR-/tmp}/am-dc-$$/" \
      && am__cwd=`pwd` \
      && CDPATH="${ZSH_VERSION+.}:" && cd dpsearch-4.53/_build \
      && ../configure \
         \
         \
        --srcdir=.. --prefix="$dc_install_base" \
      && make  \
      && make  dvi \
      && make  check \
      && make  install \
      && make  installcheck \
      && make  uninstall \
      && make  distuninstallcheck_dir="$dc_install_base" \
            distuninstallcheck \
      && chmod -R a-w "$dc_install_base" \
      && ({ \
           (cd ../.. && umask 077 && mkdir "$dc_destdir") \
           && make  DESTDIR="$dc_destdir" install \
           && make  DESTDIR="$dc_destdir" uninstall \
           && make  DESTDIR="$dc_destdir" \
                distuninstallcheck_dir="$dc_destdir" distuninstallcheck; \
          } || { rm -rf "$dc_destdir"; exit 1; }) \
      && rm -rf "$dc_destdir" \
      && make  dist \
      && rm -rf dpsearch-4.53.tar.bz2 \
      && make  distcleancheck \
      && cd "$am__cwd" \
      || exit 1
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking whether make sets $(MAKE)... (cached) yes
checking whether build environment is sane... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking how to print strings... printf
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 3458764513820540925
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for mt... mt
checking if mt is a manifest tool... no
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking whether byte ordering is bigendian... no
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for _LARGEFILE_SOURCE value needed for large files... no
checking for ANSI C header files... (cached) yes
checking for sys/types.h... (cached) yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking for unistd.h... (cached) yes
checking sys/cdefs.h usability... yes
checking sys/cdefs.h presence... yes
checking for sys/cdefs.h... yes
checking libutil.h usability... no
checking libutil.h presence... no
checking for libutil.h... no
checking sys/ioctl.h usability... yes
checking sys/ioctl.h presence... yes
checking for sys/ioctl.h... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking sys/times.h usability... yes
checking sys/times.h presence... yes
checking for sys/times.h... yes
checking sys/param.h usability... yes
checking sys/param.h presence... yes
checking for sys/param.h... yes
checking sys/wait.h usability... yes
checking sys/wait.h presence... yes
checking for sys/wait.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking sched.h usability... yes
checking sched.h presence... yes
checking for sched.h... yes
checking syslog.h usability... yes
checking syslog.h presence... yes
checking for syslog.h... yes
checking select.h usability... no
checking select.h presence... no
checking for select.h... no
checking sys/select.h usability... yes
checking sys/select.h presence... yes
checking for sys/select.h... yes
checking sys/socket.h usability... yes
checking sys/socket.h presence... yes
checking for sys/socket.h... yes
checking arpa/inet.h usability... yes
checking arpa/inet.h presence... yes
checking for arpa/inet.h... yes
checking netdb.h usability... yes
checking netdb.h presence... yes
checking for netdb.h... yes
checking sys/sysctl.h usability... yes
checking sys/sysctl.h presence... yes
checking for sys/sysctl.h... yes
checking sys/mman.h usability... yes
checking sys/mman.h presence... yes
checking for sys/mman.h... yes
checking sys/shm.h usability... yes
checking sys/shm.h presence... yes
checking for sys/shm.h... yes
checking netinet/in.h usability... yes
checking netinet/in.h presence... yes
checking for netinet/in.h... yes
checking netinet/in_systm.h usability... yes
checking netinet/in_systm.h presence... yes
checking for netinet/in_systm.h... yes
checking netinet/tcp.h usability... yes
checking netinet/tcp.h presence... yes
checking for netinet/tcp.h... yes
checking arpa/nameser.h usability... yes
checking arpa/nameser.h presence... yes
checking for arpa/nameser.h... yes
checking sys/ipc.h usability... yes
checking sys/ipc.h presence... yes
checking for sys/ipc.h... yes
checking semaphore.h usability... yes
checking semaphore.h presence... yes
checking for semaphore.h... yes
checking sys/sem.h usability... yes
checking sys/sem.h presence... yes
checking for sys/sem.h... yes
checking sys/msg.h usability... yes
checking sys/msg.h presence... yes
checking for sys/msg.h... yes
checking for netinet/ip.h... yes
checking for resolv.h... yes
checking for an ANSI C-conforming const... yes
checking for mode_t... yes
checking for off_t... yes
checking for pid_t... yes
checking for size_t... yes
checking for union semun... no
checking for socklen_t... yes
checking for tm_gmtoff in struct tm... yes
checking for strdup... yes
checking for strndup... yes
checking for strstr... yes
checking for strnlen... yes
checking for strcasecmp... yes
checking for strncasecmp... yes
checking for bzero... yes
checking for strtok_r... yes
checking for strcasestr... yes
checking for timegm... yes
checking for regcomp... yes
checking for socket... yes
checking for vsnprintf... yes
checking for snprintf... yes
checking for mergesort... no
checking for heapsort... no
checking for setproctitle... no
checking for setenv... yes
checking for putenv... yes
checking for break... no
checking for unsetenv... yes
checking for daemon... yes
checking for inet_addr... yes
checking for sqrt... no
checking for sqrt in -lm... yes
checking for libtre... yes
checking tre/regex.h usability... no
checking tre/regex.h presence... no
checking for tre/regex.h... no
checking for ares_init in -lcares... no
checking for ares_init in -lares... no
checking for getaddrinfo in -lbind... no
checking for getaddrinfo in -ldns... no
checking for EXTRACTOR_loadDefaultLibraries in -lextractor... no
checking for hstrerror... yes
checking for getaddrinfo... yes
checking for inet_net_pton... no
checking for pthread_setconcurrency function prototype in pthread.h... no
checking for pthread_yield function prototype in pthread.h... no
checking for thr_setconcurrency function prototype in thread.h... no
checking size of char*... 8
checking size of int... 4
checking size of long... 8
checking size of long long... 8
checking size of short... 2
checking size of void*... 8
checking for gethostbyname... yes
checking for socket... (cached) yes
checking for getopt location... unistd.h
checking return type of signal handlers... void
checking for POSIX signals... yes
checking for debug info... no
checking for paranoia support... no
checking for POSIX shared memory library... ok
checking for syslog support... enabled
checking for tracing support... disabled
checking for Google-like groupping... disabled
checking for multi DBAddr support... disabled
checking for take hops for the Neo PopRank calculation... no
checking for hops factor in the Neo PopRank... 0.1
checking for relevance method... full
checking for average word distance in relevency... enabled
checking for first word position in relevency... enabled
checking for word count in relevency... enabled
checking for best first word position in relevance... 13
checking for best average word position in relevance... 390
checking for best words count in relevance... 30
checking for best average distance between words in relevance... 20
checking for word distance factor in relevance... 25
checking for word less distance factor in relevance... 1.0
checking for first word position factor in relevance... 1.5
checking for first word less position factor in relevance... 0.31
checking for word count factor in relevance... 3.5
checking for word less count factor in relevance... 70.5
checking for non-uniform word count border value... 10
checking for non-uniform word count factor in relevance... 2000
checking for non-uniform words less count factor in relevance... 20
checking for relevance summands tracking... no
checking for -all-static linker flags... disabled
checking for pthreads... checking for pthreads_cflags... 
checking for pthreads_lib... 
             POSIX Threads found
checking for strerror_r... yes
checking for gmtime_r... yes
checking for gethostbyname_r... yes, and it takes 6 arguments
checking for external parsers support... yes
checking for MP3 ID3 tags support... yes
checking for file: URL schema support... yes
checking for http: (and ftp: - via proxy) URL schema support... yes
checking for ftp: URL schema support... yes
checking for news: URL schema support... yes
checking for Google profiler support... no
checking for DMALLOC memory debugger support... no
checking for Boehm's garbage collector support... no
checking for Electricfence memory debugger support... no
checking for internal memory debugger support... no
checking for internal file handlers debugger support... no
checking for ChaSen system support... no
checking for MeCab system support... no
checking for Openssl support... no
checking for zlib... yes
checking for inflate in -lz... yes
checking for aspell... yes
checking for new_aspell_config in -laspell... yes
checking for readline support... no
checking for additional character sets... 
checking for Apache module support... no
checking for cache mode only version of mod_dpsearch... no
checking for threadsless version of libdpsearch... no
checking for PHP module support... no
checking for Internationalized Domain Names support... no
checking for compatibility with old hash... no
checking for MySQL support... no
checking for PostgreSQL support... no
checking for mSQL support... no
checking for iODBC support... no
checking for unixODBC support... no
checking for IBM DB2 support... no
checking for Solid support... no
checking for OpenLink ODBC support... no
checking for EasySoft ODBC support... no
checking for SAPDB ODBC support... no
checking for InterBase support... no
checking for Ct-Lib support... no
checking for FreeTDS Ct-Lib support... no
checking for sqlite support... no
checking for sqlite3 support... no
checking for Oracle7 support... no
checking for Oracle8 support... no
checking for Oracle8i support... no
checking for Oracle Instant Client support... no
checking for openjade... openjade
checking for onsgmls... onsgmls
checking for DocBook V4.1 3.1 3.0... 4.1 yes
checking for DocBook stylesheets... /usr/share/sgml/docbook/stylesheet/dsssl/modular
checking for collateindex.pl... /usr/bin/collateindex.pl
checking that generated files are newer than configure... done
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating include/Makefile
config.status: creating src/Makefile
config.status: creating doc/Makefile
config.status: creating etc/Makefile
config.status: creating misc/Makefile
config.status: creating scripts/Makefile
config.status: creating create/Makefile
config.status: creating create/ibase/Makefile
config.status: creating create/mssql/Makefile
config.status: creating create/mysql/Makefile
config.status: creating create/oracle/Makefile
config.status: creating create/pgsql/Makefile
config.status: creating create/sqlite/Makefile
config.status: creating create/mimer/Makefile
config.status: creating scripts/dps-config
config.status: creating scripts/run-splitter
config.status: creating perl/Makefile.PL
config.status: creating test/Makefile
config.status: creating include/dps_config.h
config.status: executing depfiles commands
config.status: executing libtool commands


No database selected! indexering part will not be compiled
If it's incorrect, use --with-XXX where XXX is your preferred database type

**************************************************
*                                                *
*     Now run make && make install               *
*     If it fails, try to use GNU make,          *
*     often installed as gmake.                  *
*                                                *
*   Thank you for using DataparkSearch Engine!   *
**************************************************

make[1]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/dpsearch-4.53/_build'
Making all in include
make[2]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/dpsearch-4.53/_build/include'
make  all-am
make[3]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/dpsearch-4.53/_build/include'
make[3]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/dpsearch-4.53/_build/include'
make[2]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/dpsearch-4.53/_build/include'
Making all in src
make[2]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/dpsearch-4.53/_build/src'
Making all in .
make[3]: Betrete Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/dpsearch-4.53/_build/src'
/bin/bash ../libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../src -I../include -I../../src/../include    -DHAVE_PTHREAD -D_REENTRANT -g -O2 -DDPS_CONF_DIR=\"/home/richter/sources/dpsearch-4.53/dpsearch-4.53/_inst/etc\" -DDPS_VAR_DIR=\"/home/richter/sources/dpsearch-4.53/dpsearch-4.53/_inst/var\" -DDPS_SHARE_DIR=\"/home/richter/sources/dpsearch-4.53/dpsearch-4.53/_inst/share\" -MT libdpsearch_la-indexer.lo -MD -MP -MF .deps/libdpsearch_la-indexer.Tpo -c -o libdpsearch_la-indexer.lo `test -f 'indexer.c' || echo '../../src/'`indexer.c
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I../../src -I../include -I../../src/../include -DHAVE_PTHREAD -D_REENTRANT -g -O2 -DDPS_CONF_DIR=\"/home/richter/sources/dpsearch-4.53/dpsearch-4.53/_inst/etc\" -DDPS_VAR_DIR=\"/home/richter/sources/dpsearch-4.53/dpsearch-4.53/_inst/var\" -DDPS_SHARE_DIR=\"/home/richter/sources/dpsearch-4.53/dpsearch-4.53/_inst/share\" -MT libdpsearch_la-indexer.lo -MD -MP -MF .deps/libdpsearch_la-indexer.Tpo -c ../../src/indexer.c  -fPIC -DPIC -o .libs/libdpsearch_la-indexer.o
In file included from ../../src/../include/dps_common.h:901:0,
                 from ../../src/indexer.c:19:
../../src/../include/dps_db_int.h:82:22: fatal error: libpq-fe.h: No such file or directory
 #include "libpq-fe.h"
                      ^
compilation terminated.
make[3]: *** [libdpsearch_la-indexer.lo] Fehler 1
make[3]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/dpsearch-4.53/_build/src'
make[2]: *** [all-recursive] Fehler 1
make[2]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/dpsearch-4.53/_build/src'
make[1]: *** [all-recursive] Fehler 1
make[1]: Verlasse Verzeichnis '/mnt/sources_image_btrfs/dpsearch-4.53/dpsearch-4.53/_build'
make: *** [distcheck] Fehler 1

libpq has been installed with

wget http://ftp.postgresql.org/pub/source/v9.3.4/postgresql-9.3.4.tar.bz2
tar xf ...
./configure
cd src/interfaces/libpq
make 
sudo make install

indexing stops after a while

an indexer built with: ./configure --enable-googlegrp --enable-pophops --enable-reltrack --enable-idn --with-optimise --with-openssl --with-aspell --with-readline --with-mysql

after indexing for a while: ./sbin/indexer -o -rr -q -N 10

          Database statistics

Status    Expired      Total
   -----------------------------
     0          0     143314 Not indexed yet
   200          0        105 OK
   301          0         29 Moved Permanently
   302          0         12 Moved Temporarily
   400          0          1 Bad Request
   403          0          2 Forbidden
   404          0          9 Not found
   415          0         10 Unsupported Media Type
  2200          0          2 Clones, OK
   -----------------------------
 Total          0     143484

stops doing anything interesting: ./sbin/indexer -r -b -v 5

indexer[30554]: {00} DpsOpenCache:
indexer[30554]: {00} i:0  cached_sd:0  sin_port:0
indexer[30554]: {00} wrd_buf: 1c99b010
indexer[30554]: {00} Done.
indexer[30554]: {00} indexer from dpsearch-4.54-2017-09-05-mysql started with '/usr/local/dpsearch/etc/indexer.conf'
indexer[30554]: {00} Chinese dictionary with 0 entries
indexer[30554]: {00} Korean dictionary with 0 entries
indexer[30554]: {00} Thai dictionary with 0 entries
indexer[30554]: {00} LogsOnly: no
indexer[30554]: {00} mutexes used: 4105
indexer[30554]: {00} The following sections are defined
indexer[30554]: {00} Sections.body [1,256:00]: <NULL>
indexer[30554]: {00} Sections.Charset [12,32:00]: <NULL>
indexer[30554]: {00} Sections.Content-Language [14,16:00]: <NULL>
indexer[30554]: {00} Sections.Content-Type [13,64:00]: <NULL>
indexer[30554]: {00} Sections.crosswords [11,0:00]: <NULL>
indexer[30554]: {00} Sections.meta.description [4,128:00]: <NULL>
indexer[30554]: {00} Sections.meta.keywords [3,128:00]: <NULL>
indexer[30554]: {00} Sections.title [2,128:00]: <NULL>
indexer[30554]: {00} Sections.url [6,0:01]: <NULL>
indexer[30554]: {00} Done (55 seconds, 0 documents, 0 bytes,  0.00 Kbytes/sec.)
indexer[30554]: {00} Total 56 seconds, 0 documents, 0 bytes,  0.00 Kbytes/sec,  0.00 sec/doc, 0 bytes/doc.
indexer[30554]: {00} Neo PopRank: 0 documents, 0 pas,  0.00 Kpas/sec,  0.00 sec/doc,  0.00 pas/doc.
indexer[30554]: {00} Flushing cached buffers for mysql://xxx:xxx@xxx/xxx/?dbmode=cache... 
indexer[30554]: {00} Thu 08 18:32:17 [30554] Flushing all buffers... 
indexer[30554]: {00} Done

Help? 💯

wrong crosswords processing by indexer

Indexer is putting alt, title attributes content of IMG, A and LINK tags to crosswords section of the same document, instead of putting them as crosswords of documents linked by that tags.

Commented out code in parsehtml.c:

 else if (href && CrosSec && alt != NULL && TAG_WITH_CROSSATTRIBUTE ) {
    Item.href = href;
    Item.section = CrosSec->section;
    Item.section_name = CrosSec->name;
    Item.strict = CrosSec->strict;
    Item.str = alt;
    Item.len = dps_strlen(alt);
    putItem(Indexer, Doc, &Item);
}
 else if (href && CrosSec && title != NULL && TAG_WITH_CROSSATTRIBUTE ) {
    Item.href = href;
    Item.section = CrosSec->section;
    Item.section_name = CrosSec->name;
    Item.strict = CrosSec->strict;
    Item.str = title;
    Item.len = dps_strlen(title);
    putItem(Indexer, Doc, &Item);
}

"./configure --with-mysql=/usr/include/mysql/" fails with "configure: error: Invalid MySQL directory - unable to find mysql.h under /usr/include/mysql/" although "/usr/include/mysql/mysql.h" is present

$ stat /usr/include/mysql/mysql.h 
  File: '/usr/include/mysql/mysql.h'
  Size: 29172       Blocks: 64         IO Block: 4096   regular file
Device: 16h/22d Inode: 5569333     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2016-05-13 10:57:03.228144358 +0200
Modify: 2016-03-28 20:06:12.000000000 +0200
Change: 2016-05-04 11:08:00.626872422 +0200
 Birth: -

experienced with 9257ced

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.