intel / hyperscan Goto Github PK
View Code? Open in Web Editor NEWHigh-performance regular expression matching library
Home Page: https://www.hyperscan.io
License: Other
High-performance regular expression matching library
Home Page: https://www.hyperscan.io
License: Other
Log:
17:08:25 -- The C compiler identification is GNU 5.2.0
17:08:25 -- The CXX compiler identification is GNU 5.2.0
17:08:25 -- Check for working C compiler: /usr/local/gcc-5.2.0/bin/gcc
17:08:25 -- Check for working C compiler: /usr/local/gcc-5.2.0/bin/gcc -- works
17:08:25 -- Detecting C compiler ABI info
17:08:25 -- Detecting C compiler ABI info - done
17:08:25 -- Check for working CXX compiler: /usr/local/gcc-5.2.0/bin/g++
17:08:25 -- Check for working CXX compiler: /usr/local/gcc-5.2.0/bin/g++ -- works
17:08:25 -- Detecting CXX compiler ABI info
17:08:26 -- Detecting CXX compiler ABI info - done
17:08:26 -- Performing Test ARCH_64_BIT
17:08:26 -- Performing Test ARCH_64_BIT - Success
17:08:26 -- Performing Test ARCH_32_BIT
17:08:26 -- Performing Test ARCH_32_BIT - Failed
17:08:26 -- Default build type 'Release with debug info'
17:08:26 -- Boost version: 1.64.0
17:08:26 -- Found PythonInterp: /usr/bin/python (found version "2.4.3")
17:08:26 -- Build date: 2017-07-14
17:08:26 -- gcc version 5.2.0
17:08:26 -- g++ version 5.2.0
17:08:26 -- Looking for include file unistd.h
17:08:26 -- Looking for include file unistd.h - found
17:08:26 -- Looking for include file intrin.h
17:08:26 -- Looking for include file intrin.h - not found
17:08:26 -- Looking for C++ include intrin.h
17:08:26 -- Looking for C++ include intrin.h - not found
17:08:26 -- Looking for include file x86intrin.h
17:08:26 -- Looking for include file x86intrin.h - found
17:08:26 -- Looking for C++ include x86intrin.h
17:08:27 -- Looking for C++ include x86intrin.h - found
17:08:27 -- Looking for posix_memalign
17:08:27 -- Looking for posix_memalign - found
17:08:27 -- Looking for _aligned_malloc
17:08:27 -- Looking for _aligned_malloc - not found
17:08:27 -- Performing Test HAS_C_HIDDEN
17:08:27 -- Performing Test HAS_C_HIDDEN - Success
17:08:27 -- Performing Test HAS_CXX_HIDDEN
17:08:27 -- Performing Test HAS_CXX_HIDDEN - Success
17:08:27 -- Looking for _LIBCPP_VERSION
17:08:27 -- Looking for _LIBCPP_VERSION - not found
17:08:27 -- generator is Unix Makefiles
17:08:27 -- Performing Test HAS_C_ATTR_IFUNC
17:08:27 -- Performing Test HAS_C_ATTR_IFUNC - Success
17:08:27 -- Performing Test HAVE_SSSE3
17:08:28 -- Performing Test HAVE_SSSE3 - Success
17:08:28 -- Performing Test HAVE_AVX2
17:08:28 -- Performing Test HAVE_AVX2 - Success
17:08:28 -- Performing Test HAVE_AVX512
17:08:28 -- Performing Test HAVE_AVX512 - Failed
17:08:28 -- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED
17:08:29 -- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED - Success
17:08:29 -- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED
17:08:29 -- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED - Success
17:08:29 -- Performing Test HAVE__BUILTIN_CONSTANT_P
17:08:29 -- Performing Test HAVE__BUILTIN_CONSTANT_P - Success
17:08:29 -- Performing Test C_FLAG_Wvla
17:08:29 -- Performing Test C_FLAG_Wvla - Success
17:08:29 -- Performing Test C_FLAG_Wpointer_arith
17:08:29 -- Performing Test C_FLAG_Wpointer_arith - Success
17:08:29 -- Performing Test C_FLAG_Wstrict_prototypes
17:08:29 -- Performing Test C_FLAG_Wstrict_prototypes - Success
17:08:29 -- Performing Test C_FLAG_Wmissing_prototypes
17:08:29 -- Performing Test C_FLAG_Wmissing_prototypes - Success
17:08:29 -- Performing Test CXX_FLAG_Wvla
17:08:29 -- Performing Test CXX_FLAG_Wvla - Success
17:08:30 -- Performing Test CXX_FLAG_Wpointer_arith
17:08:30 -- Performing Test CXX_FLAG_Wpointer_arith - Success
17:08:30 -- Performing Test CC_SELF_ASSIGN
17:08:30 -- Performing Test CC_SELF_ASSIGN - Failed
17:08:30 -- Performing Test CXX_SELF_ASSIGN
17:08:30 -- Performing Test CXX_SELF_ASSIGN - Failed
17:08:30 -- Performing Test CC_PAREN_EQUALITY
17:08:30 -- Performing Test CC_PAREN_EQUALITY - Failed
17:08:30 -- Performing Test CXX_UNUSED_CONST_VAR
17:08:30 -- Performing Test CXX_UNUSED_CONST_VAR - Failed
17:08:30 -- Performing Test CXX_IGNORED_ATTR
17:08:30 -- Performing Test CXX_IGNORED_ATTR - Failed
17:08:30 -- Performing Test CXX_WEAK_VTABLES
17:08:30 -- Performing Test CXX_WEAK_VTABLES - Failed
17:08:30 -- Performing Test CXX_MISSING_DECLARATIONS
17:08:30 -- Performing Test CXX_MISSING_DECLARATIONS - Success
17:08:30 -- Performing Test CXX_WUNUSED_VARIABLE
17:08:30 -- Performing Test CXX_WUNUSED_VARIABLE - Success
17:08:30 -- Building runtime for multiple microarchitectures
17:08:30 -- Looking for mmap
17:08:30 -- Looking for mmap - found
17:08:30 -- Sphinx not found, unable to generate developer reference
17:08:30 -- Looking for include file pthread.h
17:08:30 -- Looking for include file pthread.h - found
17:08:30 -- Looking for pthread_create
17:08:30 -- Looking for pthread_create - not found
17:08:30 -- Looking for pthread_create in pthreads
17:08:30 -- Looking for pthread_create in pthreads - not found
17:08:30 -- Looking for pthread_create in pthread
17:08:30 -- Looking for pthread_create in pthread - found
17:08:30 -- Found Threads: TRUE
17:08:30 -- checking for module 'sqlite3'
17:08:30 -- found sqlite3, version 3.3
17:08:30 -- Performing Test SQLITE_VERSION_OK
17:08:31 -- Performing Test SQLITE_VERSION_OK - Success
17:08:31 -- Looking for sqlite3_open_v2
17:08:31 -- Looking for sqlite3_open_v2 - not found
17:08:31 -- Looking for C++ include pthread_np.h
17:08:31 -- Looking for C++ include pthread_np.h - not found
17:08:31 -- Looking for pthread_setaffinity_np
17:08:31 -- Looking for pthread_setaffinity_np - found
17:08:31 -- Looking for malloc_info
17:08:31 -- Looking for malloc_info - not found
17:08:31 -- Looking for shmget
17:08:31 -- Looking for shmget - found
17:08:31 -- Could not find libpcap - some examples will not be built
17:08:31 -- Configuring done
17:08:31 -- Generating done
17:08:31 -- Build files have been written to: /hyperscan-4.5.1/build
17:08:31 Scanning dependencies of target hs_exec_core2
17:08:31 [ 0%] Building C object CMakeFiles/hs_exec_core2.dir/src/crc32.c.o
17:08:31 mktemp: invalid option -- -
17:08:31 Usage: mktemp [-V] | [-dqtu] [-p prefix] [template]
17:08:31 make[2]: *** [CMakeFiles/hs_exec_core2.dir/src/crc32.c.o] Error 1
17:08:31 make[1]: *** [CMakeFiles/hs_exec_core2.dir/all] Error 2
17:08:31 make: *** [all] Error 2
As the error message, mktemp
on CentOS el5u7 doesn't support mktemp --tmpdir
but support mktemp -t
, cmake/build_wrapper.sh:14
cause the error.
Hi
In my current test with hyperscan 4.5.1 release, sometimes, it will deadloop in function roseRunProgram_i. debug with gdb, I found the value of code in "switch ((enum RoseInstructionCode)code)" is not in the range [ROSE_INSTR_END, LAST_ROSE_INSTRUCTION], and the func roseRunProgram_i has no default case, so it will deadloop by for (;;) { ............}, as the code's value has not changed.
Any one could give me a method to avoid it(not fix it in hyperscan) as soon as possible, Thanks! Thanks!
I found from https://01.org/zh/hyperscan ,
XIANG WANG
username: xiangwa1
Job Title: Software Engineer
Country: China
Could anyone give me his mailbox, or please give me a mail to [email protected], we talk with chinese is most convenient.
the gdb info as follow:
(gdb) bt
#0 0x00007f5554c18796 in roseRunProgram_i (match_len=, prog_flags=, end=, som=,
programOffset=, scratch=, t=)
at /home/jenkins_workspace/SDK-corei7-avx/hyperscan-4.5.1/src/rose/program_runtime.h:1877
#1 roseRunProgram (t=, scratch=, programOffset=, som=, end=,
match_len=, prog_flags=) at /home/jenkins_workspace/SDK-corei7-avx/hyperscan-4.5.1/src/rose/program_runtime.c:48
#2 0x00007f5554bf2f8e in roseAnchoredCallback (start=start@entry=0, end=, id=, ctx=)
at /home/jenkins_workspace/SDK-corei7-avx/hyperscan-4.5.1/src/rose/match.c:219
#3 0x00007f5554babb5b in doComplexReport (cached_accept_id=, cached_accept_state=, eod=0 '\000', loc=5,
s=10, m=0x83cf2540, ctxt=0x83cc8180, cb=0x7f5554bf2f10 )
at /home/jenkins_workspace/SDK-corei7-avx/hyperscan-4.5.1/src/nfa/mcclellan.c:74
#4 mcclellanExec8_i (mode=CALLBACK_OUTPUT, c_final=0x0, single=0 '\000', ctxt=0x83cc8180, cb=0x7f5554bf2f10 , offAdj=0,
len=5,
buf=0x7f54e987f076 "POST /router/rest?sign=2520D80952DBA4D5042FD8D22D216A42×tamp=2012-08-31+10%3A20%3A50&v=2.0&app_key=12349228&method=taobao.trade.fullinfo.get&sign_method=hmac&partner_id=top-sdk-java-20120817&sess"..., state=, m=0x83cf2540)
at /home/jenkins_workspace/SDK-corei7-avx/hyperscan-4.5.1/src/nfa/mcclellan.c:430
#5 nfaExecMcClellan8_Bi (single=0 '\000', context=0x83cc8180, cb=0x7f5554bf2f10 , length=5,
buffer=0x7f54e987f076 "POST /router/rest?sign=2520D80952DBA4D5042FD8D22D216A42×tamp=2012-08-31+10%3A20%3A50&v=2.0&app_key=12349228&method=taobao.trade.fullinfo.get&sign_method=hmac&partner_id=top-sdk-java-20120817&sess"..., offset=0, n=0x83cf2500)
at /home/jenkins_workspace/SDK-corei7-avx/hyperscan-4.5.1/src/nfa/mcclellan.c:841
#6 nfaExecMcClellan8_B (n=0x83cf2500, offset=0,
buffer=0x7f54e987f076 "POST /router/rest?sign=2520D80952DBA4D5042FD8D22D216A42×tamp=2012-08-31+10%3A20%3A50&v=2.0&app_key=12349228&method=taobao.trade.fullinfo.get&sign_method=hmac&partner_id=top-sdk-java-20120817&sess"..., length=5, cb=0x7f5554bf2f10 ,
context=0x83cc8180) at /home/jenkins_workspace/SDK-corei7-avx/hyperscan-4.5.1/src/nfa/mcclellan.c:864
#7 0x00007f5554bdd4a4 in runAnchoredTableBlock (t=, scratch=0x83cc8180, atable=)
at /home/jenkins_workspace/SDK-corei7-avx/hyperscan-4.5.1/src/rose/block.c:63
#8 roseBlockAnchored (scratch=0x83cc8180, t=0x83cf2200) at /home/jenkins_workspace/SDK-corei7-avx/hyperscan-4.5.1/src/rose/block.c:213
#9 roseBlockExec (t=, scratch=) at /home/jenkins_workspace/SDK-corei7-avx/hyperscan-4.5.1/src/rose/block.c:396
#10 0x00007f5554b2d8fe in rawBlockExec (scratch=0x83cc8180, rose=0x83cf2200)
at /home/jenkins_workspace/SDK-corei7-avx/hyperscan-4.5.1/src/runtime.c:186
#11 hs_scan (db=, data=, length=488, flags=, scratch=0x83cc8180, onEvent=,
userCtx=0x83cd0cc0) at /home/jenkins_workspace/SDK-corei7-avx/hyperscan-4.5.1/src/runtime.c:408
#12 0x00007f55551fb59d in matchTargetBlockMode (dbinfo=0x83cd0cc0,
target=0x7f54e987f076 "POST /router/rest?sign=2520D80952DBA4D5042FD8D22D216A42×tamp=2012-08-31+10%3A20%3A50&v=2.0&app_key=12349228&method=taobao.trade.fullinfo.get&sign_method=hmac&partner_id=top-sdk-java-20120817&sess"..., targetlen=488) at hyperscanwrap.cpp:536
#13 0x00007f55620b6e21 in regex_port_rule_match (coreid=, sport=, dport=,
protoc=protoc@entry=6 '\006',
---Type to continue, or q to quit---
payload=0x7f54e987f076 "POST /router/rest?sign=2520D80952DBA4D5042FD8D22D216A42×tamp=2012-08-31+10%3A20%3A50&v=2.0&app_key=12349228&method=taobao.trade.fullinfo.get&sign_method=hmac&partner_id=top-sdk-java-20120817&sess"..., paylen=488, dbinfo=dbinfo@entry=0x7f54eaf94740)
at rule_match.cpp:97
(gdb) p code
$8 = 250 '\372'
(gdb)
Hello.
I'm failed to compile hyperscan on Centos7. It's the compiler error as the following:
make[2]: *** [CMakeFiles/hs.dir/src/nfa/dfa_build_strat.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /shared/test/deps/hyperscan-4.5.0/src/nfagraph/ng_holder.h:43:0,
from /shared/test/deps/hyperscan-4.5.0/src/nfagraph/ng_repeat.h:36,
from /shared/test/deps/hyperscan-4.5.0/src/nfa/castlecompile.h:39,
from /shared/test/deps/hyperscan-4.5.0/src/nfa/castlecompile.cpp:34:
/shared/test/deps/hyperscan-4.5.0/src/util/ue2_containers.h: In instantiation of ‘void ue2::flat_set<T, Compare, Allocator>::erase(ue2::flat_set<T, Compare, Allocator>::const_iterator) [with T = unsigned int; Compare = std::less<unsigned int>; Allocator = std::allocator<unsigned int>; ue2::flat_set<T, Compare, Allocator>::const_iterator = ue2::flat_detail::iter_wrapper<__gnu_cxx::__normal_iterator<const unsigned int*, std::vector<unsigned int> >, const unsigned int>; typename ue2::flat_detail::flat_base<T, Compare, Allocator>::storage_type::const_iterator = __gnu_cxx::__normal_iterator<const unsigned int*, std::vector<unsigned int> >]’:
/shared/test/deps/hyperscan-4.5.0/src/util/ue2_containers.h:295:21: required from ‘void ue2::flat_set<T, Compare, Allocator>::erase(const key_type&) [with T = unsigned int; Compare = std::less<unsigned int>; Allocator = std::allocator<unsigned int>; ue2::flat_set<T, Compare, Allocator>::key_type = unsigned int]’
/shared/test/deps/hyperscan-4.5.0/src/nfa/castlecompile.cpp:726:27: required from here
/shared/test/deps/hyperscan-4.5.0/src/util/ue2_containers.h:285:9: error: no matching function for call to ‘std::vector<unsigned int>::erase(__gnu_cxx::__normal_iterator<const unsigned int*, std::vector<unsigned int> >)’
data().erase(pos.get());
My env is the following:
[root@localhost hyperscan-4.5.0]# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
[root@localhost hyperscan-4.5.0]# gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Any idea ?
I write some simple patterns in patterns.txt, error reported:
Pattern file: patterns.txt
'nsupported flag '
Thanks!
Building with -march=native
fails on Intel Skylake (both v4.1.0 and HEAD)
bash-3.2$ sysctl -n machdep.cpu.brand_string hw.model hw.cpufamily
Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
iMac17,1
939270559
939270559 -> 0x37fc219f -> Skylake
Full log:
https://gist.github.com/ilovezfs/867f9df96dc5b054cc99
Excerpt of the failure:
/usr/local/Library/ENV/4.3/clang -I/tmp/hyperscan20160308-37150-9e11vj/build -I/tmp/hyperscan20160308-37150-9e11vj -I/tmp/hyperscan20160308-37150-9e11vj/src -isystem /tmp/hyperscan20160308-37150-9e11vj/include -isystem /usr/local/include -I/tmp/hyperscan20160308-37150-9e11vj/build/src/fdr -O3 -std=c99 -Wall -Wextra -Wshadow -Wcast-qual -Werror -march=native -mtune=native -fvisibility=hidden -Wvla -Wpointer-arith -Wstrict-prototypes -Wmissing-prototypes -Wno-self-assign -Wno-parentheses-equality -DNDEBUG -o CMakeFiles/hs_exec.dir/src/fdr/teddy.c.o -c /tmp/hyperscan20160308-37150-9e11vj/src/fdr/teddy.c
In file included from /tmp/hyperscan20160308-37150-9e11vj/src/fdr/teddy.c:31:
/tmp/hyperscan20160308-37150-9e11vj/src/util/simd_utils_ssse3.h:37:2: error: SSSE3 instructions must be enabled
#error SSSE3 instructions must be enabled
^
/tmp/hyperscan20160308-37150-9e11vj/src/util/simd_utils_ssse3.h:73:12: error: assigning to 'm128' (aka '__m128i') from incompatible type 'int'
result = _mm_shuffle_epi8(a, b);
^ ~~~~~~~~~~~~~~~~~~~~~~
In file included from /tmp/hyperscan20160308-37150-9e11vj/src/fdr/teddy.c:244:
/tmp/hyperscan20160308-37150-9e11vj/build/src/fdr/teddy_autogen.c:14490:25: error: assigning to 'm128' (aka '__m128i') from incompatible type 'int'
res_shifted_0_1 = palignr(res_0_1, res_old_1, 16-1);
There are some issues with the build on VS 2015:
I want to detect and handle "Pattern is too large" errors. The only option I have now is to look for string "Pattern is too large" in hs_compile_error.message
, but it is a poor solution. Could you provide a enum
with error types and add it to struct hs_compile_error
, please? Description of hs_compile_error
has a list of possible problems, it can be converted to the enum
.
Using version 4.4 building on CentOS 7.
When trying to build HyperScan in a sub-directory of certain paths such as "/home/our.business.com/someuser" the regex used with sed in cmake/build_wrapper.sh mangles paths.
Error from calling 'make':
/home/business.example.com/someuser/regexengine/build/hyperscan-prefix/src/hyperscan/cmake/build_wrapper.sh: line 15: /opt/rh/devtoolset-6/root/usr/bin/ccom/someuser/regexengine/build/hyperscan-prefix/src/hyperscan-build: No such file or directory CMakeFiles/hs_exec_shared_avx2.dir/build.make:62: recipe for target 'CMakeFiles/hs_exec_shared_avx2.dir/src/crc32.c.o' failed
Output from build_wrapper.sh modified to print pre-sed and post-sed strings:
Before sed:
/opt/rh/devtoolset-6/root/usr/bin/cc -I/home/business.example.com/someuser/regexengine/build/hyperscan-prefix/src/hyperscan-build -I/home/business.example.com/someuser/regexengine/build/hyperscan-prefix/src/hyperscan -I/home/business.example.com/someuser/regexengine/build/hyperscan-prefix/src/hyperscan/src -isystem /home/business.example.com/someuser/regexengine/build/hyperscan-prefix/src/hyperscan/include -isystem /usr/local/include -O3 -std=c99 -Wall -Wextra -Wshadow -Wcast-qual -fno-strict-aliasing -DNDEBUG -Wno-array-bounds -Wno-maybe-uninitialized -fno-omit-frame-pointer -Wno-abi -fvisibility=hidden -Wvla -Wpointer-arith -Wstrict-prototypes -Wmissing-prototypes -g -DNDEBUG -fPIC -march=core2 -o CMakeFiles/hs_exec_shared_core2.dir/src/crc32.c.o -c /home/business.example.com/someuser/regexengine/build/hyperscan-prefix/src/hyperscan/src/crc32.c
After sed:
/opt/rh/devtoolset-6/root/usr/bin/ccom/someuser/regexengine/build/hyperscan-prefix/src/hyperscan-build -I/home/business.example.com/someuser/regexengine/build/hyperscan-prefix/src/hyperscan -I/home/business.example.com/someuser/regexengine/build/hyperscan-prefix/src/hyperscan/src -isystem /home/business.example.com/someuser/regexengine/build/hyperscan-prefix/src/hyperscan/include -isystem /usr/local/include -O3 -std=c99 -Wall -Wextra -Wshadow -Wcast-qual -fno-strict-aliasing -DNDEBUG -Wno-array-bounds -Wno-maybe-uninitialized -fno-omit-frame-pointer -Wno-abi -fvisibility=hidden -Wvla -Wpointer-arith -Wstrict-prototypes -Wmissing-prototypes -g -DNDEBUG -fPIC -march=core2/home/business.example.com/someuser/regexengine/build/hyperscan-prefix/src/hyperscan/src/crc32.c
Hello,
I have been developing an app with very simple function to compile a pattern and use the pattern to match which needs use Hyperscan library:
After compiling the hyperscan library with following options: -DCMAKE_BUILD_TYPE=debug -DBOOST_ROOT=${BOOST_ROOT} -DBUILD_SHARED_LIBS=1 -DCMAKE_INSTALL_PREFIX=${USR_LIB_PATH}.
The App runs as expected at my compiling machine[model name : Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz], and then I copied my app and libhs.so to another machine [model name : Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz](Same X86 64bit machine).
however it crashed every time at pattern load function when I execute the app.
The compiling machine and running machine has difference cpuinfo, not sure it has relationship with this.
Could some one can help me out. Thanks.
Here are the crash info:
rgdb.sh core.simplegrep.14441
file: compiled magic version [521] does not match with shared library magic version [524]
gdb ./simplegrep -c core.simplegrep.14441
GNU gdb (GDB) Amazon Linux (7.6.1-64.33.amzn1)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-amazon-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/david/simplegrep...done.
[New LWP 14441]
Missing separate debuginfo for /usr/lib64/libstdc++.so.6
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/0a/90c35d3174805453ea67a785446d628e298b59.debug
Missing separate debuginfo for /lib64/libgcc_s.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/00/fa2883fb47b1327397bbf167c52f51a723d013.debug
Core was generated by `./simplegrep outside ss.list'.
Program terminated with signal 4, Illegal instruction.
#0 set_range (to=111, from=111, this=0x7ffee99b7910) at /home/tidy/work/app/hyperscan/hyperscan/src/util/bitfield.h:154
154 /home/tidy/work/app/hyperscan/hyperscan/src/util/bitfield.h: No such file or directory.
Missing separate debuginfos, use: debuginfo-install glibc-2.17-106.167.amzn1.x86_64
(gdb) bt
#0 set_range (to=111, from=111, this=0x7ffee99b7910) at /home/tidy/work/app/hyperscan/hyperscan/src/util/bitfield.h:154
#1 setRange (to=111 'o', from=111 'o', this=0x7ffee99b7910) at /home/tidy/work/app/hyperscan/hyperscan/src/util/charreach.h:106
#2 CharReach (to=111 'o', from=111 'o', this=0x7ffee99b7910) at /home/tidy/work/app/hyperscan/hyperscan/src/util/charreach.h:62
#3 ue2::AsciiComponentClass::add (this=0x1c3aca0, c=111) at /home/tidy/work/app/hyperscan/hyperscan/src/parser/AsciiComponentClass.cpp:124
#4 0x00007fe4c5fddcb1 in ue2::getLiteralComponentClass (c=c@entry=111 'o', nocase=<optimized out>)
at /home/tidy/work/app/hyperscan/hyperscan/src/parser/ComponentClass.cpp:421
#5 0x00007fe4c5e0b1cc in ue2::addLiteral (currentSeq=currentSeq@entry=0x1c3ac50, c=111 'o', mode=...)
at /home/tidy/work/app/hyperscan/hyperscan/src/parser/Parser.rl:181
#6 0x00007fe4c5fe78ca in ue2::parse (c_ptr=c_ptr@entry=0x7ffee99ba6ca "outside", globalMode=...)
at /home/tidy/work/app/hyperscan/hyperscan/src/parser/Parser.rl:1823
#7 0x00007fe4c5e1b7b1 in ue2::ParsedExpression::ParsedExpression (this=0x7ffee99b8570, index_in=<optimized out>, expression=0x7ffee99ba6ca "outside", flags=2,
actionId=<optimized out>, ext=0x0) at /home/tidy/work/app/hyperscan/hyperscan/src/compiler/compiler.cpp:116
#8 0x00007fe4c5e1d0c4 in ue2::addExpression (ng=..., index=index@entry=0, expression=0x7ffee99ba6ca "outside", flags=2, ext=0x0, id=0)
at /home/tidy/work/app/hyperscan/hyperscan/src/compiler/compiler.cpp:236
#9 0x00007fe4c5e19305 in ue2::hs_compile_multi_int (expressions=expressions@entry=0x7ffee99b8a98, flags=flags@entry=0x7ffee99b8a94, ids=ids@entry=0x7ffee99b8aac,
ext=ext@entry=0x0, elements=elements@entry=1, mode=mode@entry=1, platform=platform@entry=0x0, db=db@entry=0x7ffee99b8bf0, comp_error=comp_error@entry=0x7ffee99b8bf8, g=...)
at /home/tidy/work/app/hyperscan/hyperscan/src/hs.cpp:228
#10 0x00007fe4c5e19a13 in hs_compile (expression=expression@entry=0x7ffee99ba6ca "outside", flags=flags@entry=2, mode=mode@entry=1, platform=platform@entry=0x0,
db=db@entry=0x7ffee99b8bf0, error=error@entry=0x7ffee99b8bf8) at /home/tidy/work/app/hyperscan/hyperscan/src/hs.cpp:285
#11 0x0000000000400d31 in main (argc=<optimized out>, argv=<optimized out>) at /home/tidy/exam/simplegrep.c:163
(gdb) q
Compiling machine cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 69
model name : Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
stepping : 1
microcode : 0x17
cpu MHz : 2300.092
cache size : 4096 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm ida arat epb pln pts dtherm fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt
bogomips : 4600.18
clflush size : 64
cache_alignment : 64
address sizes : 42 bits physical, 48 bits virtual
power management:
Running machine cpuinfo:
cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
stepping : 4
microcode : 0x415
cpu MHz : 2494.028
cache size : 25600 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase smep erms xsaveopt
bugs :
bogomips : 4988.05
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
-Tidy
I think the working of CMake to find the Boost dependancy needs a little fix. While requiring a given version of boost is fine, suggesting to extract it in ${CMAKE_SOURCE_DIR}/include is not working. My understanding for the "hint" BOOST_INCLUDEDIR, is that is suggests to cmake to point to the required Boost installation in that directory, but not to search for the required headers inside that directory.
The proper way definitely seems to point to the boost headers using the BOOST_ROOT variable, and passing it to the script. The script's error message should then suggest the user to pass the following variable as an argument to CMake :
cmake .. -DBOOST_ROOT=/opt/boost_1_59_0/
The result is CMake working properly and making a symbolic link inside ${CMAKE_SOURCE_DIR}/include to /opt/boost_1_59_0/
P.S: There's a typo in the error message if boost is not found : "... Either install system pacakges if available ..."
I'm working on a golang binding for Hyperscan.
After I upgrade from 4.1.0 to 4.2.0, the unit tests crashed when scan a reseted stream.
I have confirmed the same code could work with Hyperscan 4.1.0
and go version go1.6.2 darwin/amd64
Are there any changes in those API behaviors?
$go test github.com/flier/gohs/hyperscan
...
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0xb01dfacedebac1e pc=0x4350861]
runtime stack:
runtime.throw(0x4641bc0, 0x2a)
/usr/local/opt/go/libexec/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
/usr/local/opt/go/libexec/src/runtime/sigpanic_unix.go:12 +0x5a
goroutine 36 [syscall, locked to thread]:
runtime.cgocall(0x4177080, 0xc8200ba618, 0xc800000000)
/usr/local/opt/go/libexec/src/runtime/cgocall.go:123 +0x11b fp=0xc8200ba5d0 sp=0xc8200ba5a0
github.com/flier/gohs/hyperscan._Cfunc_hs_scan_stream_cgo(0x4903370, 0xc8200ba6d0, 0x5, 0x7000000, 0xc8200ba658, 0xc800000000)
??:0 +0x41 fp=0xc8200ba618 sp=0xc8200ba5d0
github.com/flier/gohs/hyperscan.hsScanStream(0x4903370, 0xc8200ba6d0, 0x5, 0x5, 0x0, 0x7000000, 0xc8200ba740, 0x0, 0x0, 0x0, ...)
/Users/flier/gocode/src/github.com/flier/gohs/hyperscan/internal.go:902 +0xaf fp=0xc8200ba678 sp=0xc8200ba618
github.com/flier/gohs/hyperscan.TestStreamScan.func1.1.1.3.1()
/Users/flier/gocode/src/github.com/flier/gohs/hyperscan/internal_test.go:637 +0x1bd fp=0xc8200ba788 sp=0xc8200ba678
github.com/smartystreets/goconvey/convey.parseAction.func1(0x4c447f0, 0xc8200f9680)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x18 fp=0xc8200ba790 sp=0xc8200ba788
github.com/smartystreets/goconvey/convey.(*context).conveyInner(0xc8200f9680, 0x4642540, 0x30, 0xc820167ab0)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x21a fp=0xc8200ba7f8 sp=0xc8200ba790
github.com/smartystreets/goconvey/convey.(*context).Convey.func1()
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:163 +0x40 fp=0xc8200ba820 sp=0xc8200ba7f8
github.com/jtolds/gls._m(0x0, 0xc82014ff40)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:70 +0x2b fp=0xc8200ba838 sp=0xc8200ba820
github.com/jtolds/gls.mark4(0x0, 0xc82014ff40)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:26 +0x2b fp=0xc8200ba850 sp=0xc8200ba838
github.com/jtolds/gls._m(0x4, 0xc82014ff40)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:72 +0x5a fp=0xc8200ba868 sp=0xc8200ba850
github.com/jtolds/gls.markS(0x4, 0xc82014ff40)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:21 +0x2b fp=0xc8200ba880 sp=0xc8200ba868
github.com/jtolds/gls.addStackTag(0x4, 0xc82014ff40)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:18 +0x37 fp=0xc8200ba898 sp=0xc8200ba880
github.com/jtolds/gls.(*ContextManager).SetValues(0xc820014560, 0xc82016adb0, 0xc82014ff40)
/Users/flier/gocode/src/github.com/jtolds/gls/context.go:98 +0x4f0 fp=0xc8200baa28 sp=0xc8200ba898
github.com/smartystreets/goconvey/convey.(*context).Convey(0xc8200f94a0, 0xc8200bac20, 0x2, 0x2)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:164 +0x36b fp=0xc8200bab20 sp=0xc8200baa28
github.com/smartystreets/goconvey/convey.Convey(0xc8200bac20, 0x2, 0x2)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/doc.go:77 +0x62 fp=0xc8200bab48 sp=0xc8200bab20
github.com/flier/gohs/hyperscan.TestStreamScan.func1.1.1.3()
/Users/flier/gocode/src/github.com/flier/gohs/hyperscan/internal_test.go:639 +0x36c fp=0xc8200bac48 sp=0xc8200bab48
github.com/smartystreets/goconvey/convey.parseAction.func1(0x4c447f0, 0xc8200f94a0)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x18 fp=0xc8200bac50 sp=0xc8200bac48
github.com/smartystreets/goconvey/convey.(*context).conveyInner(0xc8200f94a0, 0x4621b20, 0x15, 0xc8201679c0)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x21a fp=0xc8200bacb8 sp=0xc8200bac50
github.com/smartystreets/goconvey/convey.(*context).Convey.func1()
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:163 +0x40 fp=0xc8200bace0 sp=0xc8200bacb8
github.com/jtolds/gls._m(0x0, 0xc82014fec0)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:70 +0x2b fp=0xc8200bacf8 sp=0xc8200bace0
github.com/jtolds/gls.mark3(0x0, 0xc82014fec0)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:25 +0x2b fp=0xc8200bad10 sp=0xc8200bacf8
github.com/jtolds/gls._m(0x3, 0xc82014fec0)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:72 +0x5a fp=0xc8200bad28 sp=0xc8200bad10
github.com/jtolds/gls.markS(0x3, 0xc82014fec0)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:21 +0x2b fp=0xc8200bad40 sp=0xc8200bad28
github.com/jtolds/gls.addStackTag(0x3, 0xc82014fec0)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:18 +0x37 fp=0xc8200bad58 sp=0xc8200bad40
github.com/jtolds/gls.(*ContextManager).SetValues(0xc820014560, 0xc82016ac60, 0xc82014fec0)
/Users/flier/gocode/src/github.com/jtolds/gls/context.go:98 +0x4f0 fp=0xc8200baee8 sp=0xc8200bad58
github.com/smartystreets/goconvey/convey.(*context).Convey(0xc8200f9380, 0xc8200bb128, 0x2, 0x2)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:164 +0x36b fp=0xc8200bafe0 sp=0xc8200baee8
github.com/smartystreets/goconvey/convey.Convey(0xc8200bb128, 0x2, 0x2)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/doc.go:77 +0x62 fp=0xc8200bb008 sp=0xc8200bafe0
github.com/flier/gohs/hyperscan.TestStreamScan.func1.1.1()
/Users/flier/gocode/src/github.com/flier/gohs/hyperscan/internal_test.go:640 +0x737 fp=0xc8200bb190 sp=0xc8200bb008
github.com/smartystreets/goconvey/convey.parseAction.func1(0x4c447f0, 0xc8200f9380)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x18 fp=0xc8200bb198 sp=0xc8200bb190
github.com/smartystreets/goconvey/convey.(*context).conveyInner(0xc8200f9380, 0x463bd40, 0x29, 0xc820167840)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x21a fp=0xc8200bb200 sp=0xc8200bb198
github.com/smartystreets/goconvey/convey.(*context).Convey.func1()
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:163 +0x40 fp=0xc8200bb228 sp=0xc8200bb200
github.com/jtolds/gls._m(0x0, 0xc82014fda0)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:70 +0x2b fp=0xc8200bb240 sp=0xc8200bb228
github.com/jtolds/gls.mark2(0x0, 0xc82014fda0)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:24 +0x2b fp=0xc8200bb258 sp=0xc8200bb240
github.com/jtolds/gls._m(0x2, 0xc82014fda0)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:72 +0x5a fp=0xc8200bb270 sp=0xc8200bb258
github.com/jtolds/gls.markS(0x2, 0xc82014fda0)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:21 +0x2b fp=0xc8200bb288 sp=0xc8200bb270
github.com/jtolds/gls.addStackTag(0x2, 0xc82014fda0)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:18 +0x37 fp=0xc8200bb2a0 sp=0xc8200bb288
github.com/jtolds/gls.(*ContextManager).SetValues(0xc820014560, 0xc82016aba0, 0xc82014fda0)
/Users/flier/gocode/src/github.com/jtolds/gls/context.go:98 +0x4f0 fp=0xc8200bb430 sp=0xc8200bb2a0
github.com/smartystreets/goconvey/convey.(*context).Convey(0xc8200f9320, 0xc8200bb620, 0x2, 0x2)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:164 +0x36b fp=0xc8200bb528 sp=0xc8200bb430
github.com/smartystreets/goconvey/convey.Convey(0xc8200bb620, 0x2, 0x2)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/doc.go:77 +0x62 fp=0xc8200bb550 sp=0xc8200bb528
github.com/flier/gohs/hyperscan.TestStreamScan.func1.1()
/Users/flier/gocode/src/github.com/flier/gohs/hyperscan/internal_test.go:641 +0x2f0 fp=0xc8200bb648 sp=0xc8200bb550
github.com/smartystreets/goconvey/convey.parseAction.func1(0x4c447f0, 0xc8200f9320)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x18 fp=0xc8200bb650 sp=0xc8200bb648
github.com/smartystreets/goconvey/convey.(*context).conveyInner(0xc8200f9320, 0x4621b00, 0x12, 0xc820167720)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x21a fp=0xc8200bb6b8 sp=0xc8200bb650
github.com/smartystreets/goconvey/convey.(*context).Convey.func1()
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:163 +0x40 fp=0xc8200bb6e0 sp=0xc8200bb6b8
github.com/jtolds/gls._m(0x0, 0xc82014fd60)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:70 +0x2b fp=0xc8200bb6f8 sp=0xc8200bb6e0
github.com/jtolds/gls.mark1(0x0, 0xc82014fd60)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:23 +0x2b fp=0xc8200bb710 sp=0xc8200bb6f8
github.com/jtolds/gls._m(0x1, 0xc82014fd60)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:72 +0x5a fp=0xc8200bb728 sp=0xc8200bb710
github.com/jtolds/gls.markS(0x1, 0xc82014fd60)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:21 +0x2b fp=0xc8200bb740 sp=0xc8200bb728
github.com/jtolds/gls.addStackTag(0x1, 0xc82014fd60)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:18 +0x37 fp=0xc8200bb758 sp=0xc8200bb740
github.com/jtolds/gls.(*ContextManager).SetValues(0xc820014560, 0xc82016aab0, 0xc82014fd60)
/Users/flier/gocode/src/github.com/jtolds/gls/context.go:98 +0x4f0 fp=0xc8200bb8e8 sp=0xc8200bb758
github.com/smartystreets/goconvey/convey.(*context).Convey(0xc8200f92c0, 0xc8200bbad0, 0x2, 0x2)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:164 +0x36b fp=0xc8200bb9e0 sp=0xc8200bb8e8
github.com/smartystreets/goconvey/convey.Convey(0xc8200bbad0, 0x2, 0x2)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/doc.go:77 +0x62 fp=0xc8200bba08 sp=0xc8200bb9e0
github.com/flier/gohs/hyperscan.TestStreamScan.func1()
/Users/flier/gocode/src/github.com/flier/gohs/hyperscan/internal_test.go:644 +0x430 fp=0xc8200bbaf8 sp=0xc8200bba08
github.com/smartystreets/goconvey/convey.parseAction.func1(0x4c447f0, 0xc8200f92c0)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x18 fp=0xc8200bbb00 sp=0xc8200bbaf8
github.com/smartystreets/goconvey/convey.(*context).conveyInner(0xc8200f92c0, 0x46213e0, 0x17, 0xc820166340)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x21a fp=0xc8200bbb68 sp=0xc8200bbb00
github.com/smartystreets/goconvey/convey.rootConvey.func1()
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x142 fp=0xc8200bbbc0 sp=0xc8200bbb68
github.com/jtolds/gls._m(0x0, 0xc82014f280)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:70 +0x2b fp=0xc8200bbbd8 sp=0xc8200bbbc0
github.com/jtolds/gls.markS(0x0, 0xc82014f280)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:21 +0x2b fp=0xc8200bbbf0 sp=0xc8200bbbd8
github.com/jtolds/gls.addStackTag(0x0, 0xc82014f280)
/Users/flier/gocode/src/github.com/jtolds/gls/stack_tags.go:18 +0x37 fp=0xc8200bbc08 sp=0xc8200bbbf0
github.com/jtolds/gls.(*ContextManager).SetValues(0xc820014560, 0xc82016a030, 0xc82014f280)
/Users/flier/gocode/src/github.com/jtolds/gls/context.go:98 +0x4f0 fp=0xc8200bbd98 sp=0xc8200bbc08
github.com/smartystreets/goconvey/convey.rootConvey(0xc8200bbf10, 0x3, 0x3)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x3eb fp=0xc8200bbe70 sp=0xc8200bbd98
github.com/smartystreets/goconvey/convey.Convey(0xc8200bbf10, 0x3, 0x3)
/Users/flier/gocode/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x45 fp=0xc8200bbe98 sp=0xc8200bbe70
github.com/flier/gohs/hyperscan.TestStreamScan(0xc8200b2b40)
/Users/flier/gocode/src/github.com/flier/gohs/hyperscan/internal_test.go:647 +0x1ae fp=0xc8200bbf48 sp=0xc8200bbe98
testing.tRunner(0xc8200b2b40, 0x47593a8)
/usr/local/opt/go/libexec/src/testing/testing.go:473 +0x98 fp=0xc8200bbf80 sp=0xc8200bbf48
runtime.goexit()
/usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8200bbf88 sp=0xc8200bbf80
created by testing.RunTests
/usr/local/opt/go/libexec/src/testing/testing.go:582 +0x892
goroutine 1 [chan receive]:
testing.RunTests(0x4658a88, 0x47591e0, 0x18, 0x18, 0xc820068f01)
/usr/local/opt/go/libexec/src/testing/testing.go:583 +0x8d2
testing.(*M).Run(0xc820089ef8, 0x400b567)
/usr/local/opt/go/libexec/src/testing/testing.go:515 +0x81
main.main()
github.com/flier/gohs/hyperscan/_test/_testmain.go:100 +0x117
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1998 +0x1
In working on getting hyperscan packaged for Fedora/EPEL we came across the following rpmlint warnings:
hyperscan-debuginfo.x86_64: W: spurious-executable-perm /usr/src/debug/hyperscan-4.3.1/src/nfa/dfa_build_strat.cpp
hyperscan-debuginfo.x86_64: W: spurious-executable-perm /usr/src/debug/hyperscan-4.3.1/src/nfa/accel_dfa_build_strat.cpp
hyperscan-debuginfo.x86_64: W: spurious-executable-perm /usr/src/debug/hyperscan-4.3.1/src/nfa/accel_dfa_build_strat.h
The files above are indeed marked as executable (0775).
Would you like to replace any double quotes by angle brackets around file names for include statements?
I need the matched startoff position , and SOM be on. So I should use block mode? Hyperscan seems to main the stream mode. My aim is to scan packets in order, better to support shard message, but it's not necessary.
We encountered some build errors with the current stable version of hyperscan 4.2.0 after Debian switched to gcc 6.1.1 lately:
[ 17%] Building CXX object CMakeFiles/hs_shared.dir/src/hs.cpp.o
/usr/bin/c++ -Dhs_shared_EXPORTS -I/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu -I/<<PKGBUILDDIR>> -I/<<PKGBUILDDIR>>/src -isystem /<<PKGBUILDDIR>>/include -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fPIE -fstack-protector-strong -Wformat -Werror=format-security -mssse3 -march=core2 -Wdate-time -D_FORTIFY_SOURCE=2 -O2 -std=c++11 -Wall -Wextra -Wno-shadow -Wswitch -Wreturn-type -Wcast-qual -Wno-deprecated -Wnon-virtual-dtor -Werror -fabi-version=0 -Wno-unused-local-typedefs -Wno-maybe-uninitialized -fno-omit-frame-pointer -Wvla -Wpointer-arith -Wno-unused-const-variable -Wmissing-declarations -fPIC -o CMakeFiles/hs_shared.dir/src/hs.cpp.o -c /<<PKGBUILDDIR>>/src/hs.cpp
In file included from /<<PKGBUILDDIR>>/src/util/charreach.h:40:0,
from /<<PKGBUILDDIR>>/src/nfagraph/ng_graph.h:41,
from /<<PKGBUILDDIR>>/src/nfagraph/ng_holder.h:32,
from /<<PKGBUILDDIR>>/src/nfagraph/ng.h:36,
from /<<PKGBUILDDIR>>/src/hs.cpp:40:
/<<PKGBUILDDIR>>/src/util/bitfield.h:420:38: error: ignoring attributes on template argument 'ue2::bitfield<requested_size>::block_type {aka long long unsigned int}' [-Werror=ignored-attributes]
std::array<block_type, num_blocks> bits;
^
In file included from /<<PKGBUILDDIR>>/src/nfagraph/ng.h:45:0,
from /<<PKGBUILDDIR>>/src/hs.cpp:40:
/<<PKGBUILDDIR>>/src/util/report_manager.h:146:23: error: ignoring attributes on template argument 's64a {aka long long int}' [-Werror=ignored-attributes]
std::map<s64a, u32> toExhaustibleKeyMap;
^
cc1plus: all warnings being treated as errors
CMakeFiles/hs_shared.dir/build.make:94: recipe for target 'CMakeFiles/hs_shared.dir/src/hs.cpp.o' failed
Building was done with the latest gcc 6.1.1 in Debian unstable:
gcc (Debian 6.1.1-11) 6.1.1 20160802
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
I will test the current development branch to see if the problem will be relevant for 4.3.0
Hi all,
Please help to take a look at and give some tips.
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /root/contentscan...done.
[New LWP 21986]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./contentscan -f /usr/local/holonet/contentscan/config.json -n 1'.
Program terminated with signal 4, Illegal instruction.
#0 0x00007fa3dd531303 in ue2::shortcutLiteral (ng=..., expr=...) at /home/tidy/work/AIE/hyperscan/hyperscan/src/parser/shortcut_literal.cpp:180
180 /home/tidy/work/AIE/hyperscan/hyperscan/src/parser/shortcut_literal.cpp: No such file or directory.
Missing separate debuginfos, use: debuginfo-install glibc-2.17-105.el7.x86_64 glibc-2.17-157.el7_3.1.x86_64 libgcc-4.8.5-11.el7.x86_64 libgcc-4.8.5-4.el7.x86_64 libstdc++-4.8.5-11.el7.x86_64 libstdc++-4.8.5-4.el7.x86_64
(gdb) bt full
#0 0x00007fa3dd531303 in ue2::shortcutLiteral (ng=..., expr=...) at /home/tidy/work/AIE/hyperscan/hyperscan/src/parser/shortcut_literal.cpp:180
PRETTY_FUNCTION = "bool ue2::shortcutLiteral(ue2::NG&, const ue2::ParsedExpression&)"
vis = {ue2::ConstComponentVisitor = {_vptr.ConstComponentVisitor = 0x7fa3ddca8330 <vtable for ue2::ConstructLiteralVisitor+16>}, lit = {s = "",
nocase = std::vector of length 0, capacity 0}, repeat_stack = std::stack wrapping: std::deque with 0 elements}
lit =
#1 0x00007fa3dd33975d in ue2::addExpression (ng=..., index=index@entry=0, expression=0x2a5c358 "\b\d{3}[-|\s]?\d{2}[-|\s]?\d{4}\b", flags=12, ext=0x0,
id=) at /home/tidy/work/AIE/hyperscan/hyperscan/src/compiler/compiler.cpp:272
PRETTY_FUNCTION = "void ue2::addExpression(ue2::NG&, unsigned int, const char*, unsigned int, const hs_expr_ext*, ReportID)"
cc = @0x7ffe878a8e60: {streaming = false, vectored = false, target_info = {tune = 4, cpu_features = 0}, grey = {optimiseComponentTree = true,
performGraphSimplification = true, prefilterReductions = true, removeEdgeRedundancy = true, allowGough = true, allowHaigLit = true, allowLitHaig = true,
allowLbr = true, allowMcClellan = true, allowSheng = true, allowMcSheng = true, allowPuff = true, allowLiteral = true, allowRose = true, allowViolet = true,
allowExtendedNFA = true, allowLimExNFA = true, allowAnchoredAcyclic = true, allowSmallLiteralSet = true, allowCastle = true, allowDecoratedLiteral = true,
allowNoodle = true, fdrAllowTeddy = true, violetAvoidSuffixes = 1, violetAvoidWeakInfixes = true, violetDoubleCut = true, violetExtractStrongLiterals = true,
violetLiteralChains = true, violetDoubleCutLiteralLen = 3, violetEarlyCleanLiteralLen = 6, puffImproveHead = true, castleExclusive = true, mergeSEP = true,
mergeRose = true, mergeSuffixes = true, mergeOutfixes = true, onlyOneOutfix = false, allowShermanStates = true, allowMcClellan8 = true, highlanderPruneDFA = true,
minimizeDFA = true, accelerateDFA = true, accelerateNFA = true, reverseAccelerate = true, squashNFA = true, compressNFAState = true, numberNFAStatesWrong = false,
highlanderSquash = true, allowZombies = true, floodAsPuffette = false, nfaForceSize = 0, maxHistoryAvailable = 110, minHistoryAvailable = 0, maxAnchoredRegion = 63,
minRoseLiteralLength = 3, minRoseNetflowLiteralLength = 2, maxRoseNetflowEdges = 50000, minExtBoundedRepeatSize = 32, goughCopyPropagate = true,
goughRegisterAllocate = true, shortcutLiterals = true, roseGraphReduction = true, roseRoleAliasing = true, roseMasks = true, roseMaxBadLeafLength = 5,
roseConvertInfBadLeaves = true, roseConvertFloodProneSuffixes = true, roseMergeRosesDuringAliasing = true, roseMultiTopRoses = true, roseHamsterMasks = true,
roseLookaroundMasks = true, roseMcClellanPrefix = 1, roseMcClellanSuffix = 1, roseMcClellanOutfix = 2, roseTransformDelay = true, roseDesiredSplit = 4,
earlyMcClellanPrefix = true, earlyMcClellanInfix = true, earlyMcClellanSuffix = true, allowCountingMiracles = true, allowSomChain = true, somMaxRevNfaLength = 126,
hamsterAccelForward = true, hamsterAccelReverse = false, miracleHistoryBonus = 16, equivalenceEnable = true, allowSmallWrite = true, allowSmallWriteSheng = false,
smallWriteLargestBuffer = 70, smallWriteLargestBufferBad = 35, limitSmallWriteOutfixSize = 1048576, smallWriteMaxPatterns = 10000, smallWriteMaxLiterals = 10000,
allowTamarama = true, tamaChunkSize = 100, dumpFlags = 0, dumpPath = "", limitPatternCount = 8000000, limitPatternLength = 16000, limitGraphVertices = 500000,
limitGraphEdges = 1000000, limitReportCount = 32000000, limitLiteralCount = 8000000, limitLiteralLength = 16000, limitLiteralMatcherChars = 1073741824,
limitLiteralMatcherSize = 1073741824, limitRoseRoleCount = 32000000, limitRoseEngineCount = 8000000, limitRoseAnchoredSize = 1073741824,
limitEngineSize = 1073741824, limitDFASize = 1073741824, limitNFASize = 1048576, limitLBRSize = 1048576}}
expr = {boost::noncopyable_::noncopyable = {}, utf8 = false, component = std::unique_ptrue2::Component containing 0x2a5c500, allow_vacuous = false,
highlander = true, prefilter = false, som = SOM_NONE, index = 0, id = 1025024, min_offset = 0, max_offset = 18446744073709551615, min_length = 0}
g = std::unique_ptrue2::NGWrapper containing 0x7ffe878a8a70
#2 0x00007fa3dd33420d in ue2::hs_compile_multi_int (expressions=expressions@entry=0x2a9c000, flags=flags@entry=0x2a5e280, ids=ids@entry=0x2a5e100, ext=ext@entry=0x0,
elements=elements@entry=20, mode=mode@entry=1, platform=platform@entry=0x0, db=db@entry=0x7ffe878a9270, comp_error=comp_error@entry=0x7ffe878a9278, g=...)
at /home/tidy/work/AIE/hyperscan/hyperscan/src/hs.cpp:237
i = 0
length = 3707816374
out =
PRETTY_FUNCTION = "hs_error_t ue2::hs_compile_multi_int(const char* const*, const unsigned int*, const unsigned int*, const hs_expr_ext* const*, unsigned int, unsigned int, const hs_platform_info_t*, hs_database_t**, hs"...
isVectored = false
cc = {streaming = false, vectored = false, target_info = {tune = 4, cpu_features = 0}, grey = {optimiseComponentTree = true, performGraphSimplification = true,
prefilterReductions = true, removeEdgeRedundancy = true, allowGough = true, allowHaigLit = true, allowLitHaig = true, allowLbr = true, allowMcClellan = true,
allowSheng = true, allowMcSheng = true, allowPuff = true, allowLiteral = true, allowRose = true, allowViolet = true, allowExtendedNFA = true, allowLimExNFA = true,
allowAnchoredAcyclic = true, allowSmallLiteralSet = true, allowCastle = true, allowDecoratedLiteral = true, allowNoodle = true, fdrAllowTeddy = true,
violetAvoidSuffixes = 1, violetAvoidWeakInfixes = true, violetDoubleCut = true, violetExtractStrongLiterals = true, violetLiteralChains = true,
violetDoubleCutLiteralLen = 3, violetEarlyCleanLiteralLen = 6, puffImproveHead = true, castleExclusive = true, mergeSEP = true, mergeRose = true,
mergeSuffixes = true, mergeOutfixes = true, onlyOneOutfix = false, allowShermanStates = true, allowMcClellan8 = true, highlanderPruneDFA = true, minimizeDFA = true,
accelerateDFA = true, accelerateNFA = true, reverseAccelerate = true, squashNFA = true, compressNFAState = true, numberNFAStatesWrong = false,
highlanderSquash = true, allowZombies = true, floodAsPuffette = false, nfaForceSize = 0, maxHistoryAvailable = 110, minHistoryAvailable = 0, maxAnchoredRegion = 63,
minRoseLiteralLength = 3, minRoseNetflowLiteralLength = 2, maxRoseNetflowEdges = 50000, minExtBoundedRepeatSize = 32, goughCopyPropagate = true,
---Type to continue, or q to quit---
goughRegisterAllocate = true, shortcutLiterals = true, roseGraphReduction = true, roseRoleAliasing = true, roseMasks = true, roseMaxBadLeafLength = 5,
roseConvertInfBadLeaves = true, roseConvertFloodProneSuffixes = true, roseMergeRosesDuringAliasing = true, roseMultiTopRoses = true, roseHamsterMasks = true,
roseLookaroundMasks = true, roseMcClellanPrefix = 1, roseMcClellanSuffix = 1, roseMcClellanOutfix = 2, roseTransformDelay = true, roseDesiredSplit = 4,
earlyMcClellanPrefix = true, earlyMcClellanInfix = true, earlyMcClellanSuffix = true, allowCountingMiracles = true, allowSomChain = true, somMaxRevNfaLength = 126,
hamsterAccelForward = true, hamsterAccelReverse = false, miracleHistoryBonus = 16, equivalenceEnable = true, allowSmallWrite = true, allowSmallWriteSheng = false,
smallWriteLargestBuffer = 70, smallWriteLargestBufferBad = 35, limitSmallWriteOutfixSize = 1048576, smallWriteMaxPatterns = 10000, smallWriteMaxLiterals = 10000,
allowTamarama = true, tamaChunkSize = 100, dumpFlags = 0, dumpPath = "", limitPatternCount = 8000000, limitPatternLength = 16000, limitGraphVertices = 500000,
limitGraphEdges = 1000000, limitReportCount = 32000000, limitLiteralCount = 8000000, limitLiteralLength = 16000, limitLiteralMatcherChars = 1073741824,
limitLiteralMatcherSize = 1073741824, limitRoseRoleCount = 32000000, limitRoseEngineCount = 8000000, limitRoseAnchoredSize = 1073741824,
limitEngineSize = 1073741824, limitDFASize = 1073741824, limitNFASize = 1048576, limitLBRSize = 1048576}}
isStreaming =
target_info = {tune = 4, cpu_features = 0}
ng = {boost::noncopyable_::noncopyable = {}, maxSomRevHistoryAvailable = 126, minWidth = {static val_infinity = 2147483647,
static val_unreachable = 2147483648, val = 2147483647}, rm = {boost::noncopyable_::noncopyable = {}, grey = @0x7ffe878a8b38,
reportIds = std::vector of length 0, capacity 0, reportIdToInternalMap = std::map with 0 elements, reportIdToDedupeKey = std::map with 0 elements,
reportIdToProgramOffset = std::map with 0 elements, externalIdMap = std::map with 0 elements, toExhaustibleKeyMap = std::map with 0 elements, freeEIndex = 0,
global_exhaust = true}, ssm = {boost::noncopyable_::noncopyable = {}, static NO_PARENT = 4294967295, nextSomSlot = 0, cache =
std::unique_ptrue2::SlotCache containing 0x2a5e180, rev_nfas = std::deque with 0 elements, historyRequired = 0, precision = 0},
boundary = {boost::noncopyable_::noncopyable = {}, report_at_0 = std::set with 0 elements, report_at_0_eod = std::set with 0 elements,
report_at_eod = std::set with 0 elements}, cc = {streaming = false, vectored = false, target_info = {tune = 4, cpu_features = 0}, grey = {
optimiseComponentTree = true, performGraphSimplification = true, prefilterReductions = true, removeEdgeRedundancy = true, allowGough = true, allowHaigLit = true,
allowLitHaig = true, allowLbr = true, allowMcClellan = true, allowSheng = true, allowMcSheng = true, allowPuff = true, allowLiteral = true, allowRose = true,
allowViolet = true, allowExtendedNFA = true, allowLimExNFA = true, allowAnchoredAcyclic = true, allowSmallLiteralSet = true, allowCastle = true,
allowDecoratedLiteral = true, allowNoodle = true, fdrAllowTeddy = true, violetAvoidSuffixes = 1, violetAvoidWeakInfixes = true, violetDoubleCut = true,
violetExtractStrongLiterals = true, violetLiteralChains = true, violetDoubleCutLiteralLen = 3, violetEarlyCleanLiteralLen = 6, puffImproveHead = true,
castleExclusive = true, mergeSEP = true, mergeRose = true, mergeSuffixes = true, mergeOutfixes = true, onlyOneOutfix = false, allowShermanStates = true,
allowMcClellan8 = true, highlanderPruneDFA = true, minimizeDFA = true, accelerateDFA = true, accelerateNFA = true, reverseAccelerate = true, squashNFA = true,
compressNFAState = true, numberNFAStatesWrong = false, highlanderSquash = true, allowZombies = true, floodAsPuffette = false, nfaForceSize = 0,
maxHistoryAvailable = 110, minHistoryAvailable = 0, maxAnchoredRegion = 63, minRoseLiteralLength = 3, minRoseNetflowLiteralLength = 2,
maxRoseNetflowEdges = 50000, minExtBoundedRepeatSize = 32, goughCopyPropagate = true, goughRegisterAllocate = true, shortcutLiterals = true,
roseGraphReduction = true, roseRoleAliasing = true, roseMasks = true, roseMaxBadLeafLength = 5, roseConvertInfBadLeaves = true,
roseConvertFloodProneSuffixes = true, roseMergeRosesDuringAliasing = true, roseMultiTopRoses = true, roseHamsterMasks = true, roseLookaroundMasks = true,
roseMcClellanPrefix = 1, roseMcClellanSuffix = 1, roseMcClellanOutfix = 2, roseTransformDelay = true, roseDesiredSplit = 4, earlyMcClellanPrefix = true,
earlyMcClellanInfix = true, earlyMcClellanSuffix = true, allowCountingMiracles = true, allowSomChain = true, somMaxRevNfaLength = 126, hamsterAccelForward = true,
hamsterAccelReverse = false, miracleHistoryBonus = 16, equivalenceEnable = true, allowSmallWrite = true, allowSmallWriteSheng = false,
smallWriteLargestBuffer = 70, smallWriteLargestBufferBad = 35, limitSmallWriteOutfixSize = 1048576, smallWriteMaxPatterns = 10000, smallWriteMaxLiterals = 10000,
allowTamarama = true, tamaChunkSize = 100, dumpFlags = 0, dumpPath = "", limitPatternCount = 8000000, limitPatternLength = 16000, limitGraphVertices = 500000,
limitGraphEdges = 1000000, limitReportCount = 32000000, limitLiteralCount = 8000000, limitLiteralLength = 16000, limitLiteralMatcherChars = 1073741824,
limitLiteralMatcherSize = 1073741824, limitRoseRoleCount = 32000000, limitRoseEngineCount = 8000000, limitRoseAnchoredSize = 1073741824,
limitEngineSize = 1073741824, limitDFASize = 1073741824, limitNFASize = 1048576, limitLBRSize = 1048576}}, smwr =
std::unique_ptr<ue2::SmallWriteBuild> containing 0x2a5c480, rose = std::unique_ptr<ue2::RoseBuild> containing 0x2aa0000}
#3 0x00007fa3dd334acc in hs_compile_multi (expressions=0x2a9c000, flags=0x2a5e280, ids=0x2a5e100, elements=20, mode=1, platform=0x0, db=0x7ffe878a9270, error=0x7ffe878a9278)
at /home/tidy/work/AIE/hyperscan/hyperscan/src/hs.cpp:305
Hello, would you please explain the relation hs_scratch_t and hs_database_t?
Before hs_scan, we should create a database, and call hs_alloc_scratch
to create scratch for every thread.
Here, my question is:
if in one thread, I have three databases, should I create three scratches? or can I share one scratch for all these databases?
Expect your answer, thanks!
The Compilation part of the tutorial provides two modifiers with the same value:
(?s)
and (?-s)
(?s)
and (?-s)
This seems to be a typo.
Could you explain what is the difference between the modifiers Dot-all, Extended syntax and Multi-line and provide examples in the documentation, please?
Performance is not very ideal when I used over 5000 regular patterns testing on the Intel Xeon (R) (R) CPU E5-2667 v3 @ 3.20 GHz, open 16 threads test
SOM off 10.2 Gbps
SOM open 3.0 Gbps
Maybe my patterns are not well-written, Could you give some grammar specification or instructions in detail
After upgrading Boost in homebrew/core from 1.61 to 1.62, hyperscan now fails to build.
The error is
[ 39%] Building CXX object CMakeFiles/hs.dir/src/nfagraph/ng_dominators.cpp.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -I/tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/build -I/tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1 -I/tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src -isystem /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/include -isystem /usr/local/include -Os -w -pipe -march=native -mmacosx-version-min=10.11 -DNDEBUG -std=c++11 -Wall -Wextra -Wshadow -Wswitch -Wreturn-type -Wcast-qual -Wno-deprecated -Wnon-virtual-dtor -fno-strict-aliasing -fno-omit-frame-pointer -fvisibility=hidden -Wvla -Wpointer-arith -Wno-self-assign -Wno-unused-const-variable -Wno-ignored-attributes -Wweak-vtables -Wmissing-declarations -O2 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -mmacosx-version-min=10.11 -o CMakeFiles/hs.dir/src/nfagraph/ng_dominators.cpp.o -c /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp
In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp:34:
In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.h:38:
In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_holder.h:32:
In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_graph.h:46:
In file included from /usr/local/include/boost/graph/adjacency_list.hpp:246:
In file included from /usr/local/include/boost/graph/detail/adjacency_list.hpp:32:
/usr/local/include/boost/graph/graph_concepts.hpp:131:13: error: no matching function for call to 'degree'
n = degree(v, g);
^~~~~~
/usr/local/include/boost/concept/usage.hpp:16:43: note: in instantiation of member function 'boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> >::~BidirectionalGraph' requested here
~usage_requirements() { ((Model*)0)->~Model(); }
^
/usr/local/include/boost/concept/detail/general.hpp:39:42: note: in instantiation of member function 'boost::concepts::usage_requirements<boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> > >::~usage_requirements' requested here
static void failed() { ((Model*)0)->~Model(); }
^
/usr/local/include/boost/graph/graph_concepts.hpp:122:9: note: in instantiation of member function 'boost::concepts::requirement<boost::concepts::failed ************boost::concepts::usage_requirements<boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> > >::************>::failed' requested here
BOOST_CONCEPT_USAGE(BidirectionalGraph) {
^
/usr/local/include/boost/concept/usage.hpp:29:7: note: expanded from macro 'BOOST_CONCEPT_USAGE'
BOOST_CONCEPT_ASSERT((boost::concepts::usage_requirements<model>)); \
^
/usr/local/include/boost/concept/assert.hpp:43:5: note: expanded from macro 'BOOST_CONCEPT_ASSERT'
BOOST_CONCEPT_ASSERT_FN(void(*)ModelInParens)
^
/usr/local/include/boost/concept/detail/general.hpp:71:51: note: expanded from macro 'BOOST_CONCEPT_ASSERT_FN'
&::boost::concepts::requirement_<ModelFnPtr>::failed> \
^
/tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp:67:16: note: in instantiation of function template specialization 'boost_ue2::lengauer_tarjan_dominator_tree<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &>, boost::adj_list_vertex_property_map<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, unsigned int, const unsigned int &, unsigned int ue2::NFAGraphVertexProps::*>, boost::iterator_property_map<std::__1::__wrap_iter<unsigned long *>, boost::adj_list_vertex_property_map<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, unsigned int, const unsigned int &, unsigned int ue2::NFAGraphVertexProps::*>, unsigned long, unsigned long &>, boost::iterator_property_map<std::__1::__wrap_iter<void **>, boost::adj_list_vertex_property_map<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, unsigned int, const unsigned int &, unsigned int ue2::NFAGraphVertexProps::*>, void *, void *&>, std::__1::vector<void *, std::__1::allocator<void *> >, boost::associative_property_map<boost::unordered::unordered_map<void *, void *, boost::hash<void *>, std::__1::equal_to<void *>, std::__1::allocator<std::__1::pair<void *const, void *> > > > >' requested here
boost_ue2::lengauer_tarjan_dominator_tree(g, source, index_map, dfnum_map,
^
/tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp:81:12: note: in instantiation of function template specialization 'ue2::calcDominators<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> >' requested here
return calcDominators(boost::reverse_graph<NFAGraph, const NFAGraph &>(g.g),
^
/usr/local/include/boost/graph/detail/adjacency_list.hpp:1087:5: note: candidate template ignored: could not match 'undirected_graph_helper' against 'reverse_graph'
degree(typename Config::vertex_descriptor u,
^
/usr/local/include/boost/graph/detail/adjacency_list.hpp:1514:5: note: candidate template ignored: could not match 'bidirectional_graph_helper_with_property' against 'reverse_graph'
degree(typename Config::vertex_descriptor u,
^
In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp:34:
In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.h:38:
In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_holder.h:32:
In file included from /tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_graph.h:46:
In file included from /usr/local/include/boost/graph/adjacency_list.hpp:246:
In file included from /usr/local/include/boost/graph/detail/adjacency_list.hpp:32:
/usr/local/include/boost/graph/graph_concepts.hpp:138:13: error: no matching function for call to 'degree'
n = degree(v, cg);
^~~~~~
/usr/local/include/boost/graph/graph_concepts.hpp:133:9: note: in instantiation of member function 'boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> >::const_constraints' requested here
const_constraints(g);
^
/usr/local/include/boost/concept/usage.hpp:16:43: note: in instantiation of member function 'boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> >::~BidirectionalGraph' requested here
~usage_requirements() { ((Model*)0)->~Model(); }
^
/usr/local/include/boost/concept/detail/general.hpp:39:42: note: in instantiation of member function 'boost::concepts::usage_requirements<boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> > >::~usage_requirements' requested here
static void failed() { ((Model*)0)->~Model(); }
^
/usr/local/include/boost/graph/graph_concepts.hpp:122:9: note: in instantiation of member function 'boost::concepts::requirement<boost::concepts::failed ************boost::concepts::usage_requirements<boost::concepts::BidirectionalGraph<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> > >::************>::failed' requested here
BOOST_CONCEPT_USAGE(BidirectionalGraph) {
^
/usr/local/include/boost/concept/usage.hpp:29:7: note: expanded from macro 'BOOST_CONCEPT_USAGE'
BOOST_CONCEPT_ASSERT((boost::concepts::usage_requirements<model>)); \
^
/usr/local/include/boost/concept/assert.hpp:43:5: note: expanded from macro 'BOOST_CONCEPT_ASSERT'
BOOST_CONCEPT_ASSERT_FN(void(*)ModelInParens)
^
/usr/local/include/boost/concept/detail/general.hpp:71:51: note: expanded from macro 'BOOST_CONCEPT_ASSERT_FN'
&::boost::concepts::requirement_<ModelFnPtr>::failed> \
^
/tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp:67:16: note: in instantiation of function template specialization 'boost_ue2::lengauer_tarjan_dominator_tree<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &>, boost::adj_list_vertex_property_map<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, unsigned int, const unsigned int &, unsigned int ue2::NFAGraphVertexProps::*>, boost::iterator_property_map<std::__1::__wrap_iter<unsigned long *>, boost::adj_list_vertex_property_map<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, unsigned int, const unsigned int &, unsigned int ue2::NFAGraphVertexProps::*>, unsigned long, unsigned long &>, boost::iterator_property_map<std::__1::__wrap_iter<void **>, boost::adj_list_vertex_property_map<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, unsigned int, const unsigned int &, unsigned int ue2::NFAGraphVertexProps::*>, void *, void *&>, std::__1::vector<void *, std::__1::allocator<void *> >, boost::associative_property_map<boost::unordered::unordered_map<void *, void *, boost::hash<void *>, std::__1::equal_to<void *>, std::__1::allocator<std::__1::pair<void *const, void *> > > > >' requested here
boost_ue2::lengauer_tarjan_dominator_tree(g, source, index_map, dfnum_map,
^
/tmp/hyperscan-20161015-47492-up8wy8/hyperscan-4.3.1/src/nfagraph/ng_dominators.cpp:81:12: note: in instantiation of function template specialization 'ue2::calcDominators<boost::reverse_graph<boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS>, const boost::adjacency_list<boost::listS, boost::listS, boost::bidirectionalS, ue2::NFAGraphVertexProps, ue2::NFAGraphEdgeProps, boost::no_property, boost::listS> &> >' requested here
return calcDominators(boost::reverse_graph<NFAGraph, const NFAGraph &>(g.g),
^
/usr/local/include/boost/graph/detail/adjacency_list.hpp:1087:5: note: candidate template ignored: could not match 'undirected_graph_helper' against 'reverse_graph'
degree(typename Config::vertex_descriptor u,
^
/usr/local/include/boost/graph/detail/adjacency_list.hpp:1514:5: note: candidate template ignored: could not match 'bidirectional_graph_helper_with_property' against 'reverse_graph'
degree(typename Config::vertex_descriptor u,
^
2 errors generated.
make[2]: *** [CMakeFiles/hs.dir/src/nfagraph/ng_dominators.cpp.o] Error 1
make[1]: *** [CMakeFiles/hs.dir/all] Error 2
make: *** [all] Error 2
/usr/local/Homebrew/Library/Homebrew/debrew.rb:11:in `raise'
Full build log is here: https://gist.github.com/ilovezfs/3b8db4415d055a04cf0099e585ad12f3
I build hyperscan in shared mode:
$ cmake ../hyperscan-4.1.0 -DBUILD_SHARED_LIBS=ON
$ make
It fails to build bin/unit-hyperscan
because of linking errors:
allocators.cpp:41: undefined reference to `testing::Message::Message()'
allocators.cpp:41: undefined reference to `testing::internal::GetBoolAssertionFailureMessage(testing::AssertionResult const&, char const*, char const*, char const*)'
allocators.cpp:41: undefined reference to `testing::internal::AssertHelper::AssertHelper(testing::TestPartResult::Type, char const*, int, char const*)'
allocators.cpp:41: undefined reference to `testing::internal::AssertHelper::operator=(testing::Message const&) const'
allocators.cpp:41: undefined reference to `testing::internal::AssertHelper::~AssertHelper()'
... and so on
It is caused by the fact that symbols in libgtest.so are local:
$ nm --demangle lib/libgtest.so | grep IsTrue
000000000000a810 t testing::internal::IsTrue(bool)
It should be T
instead of t
there.
This problem seems to be caused by using -fvisibility=hidden
compiler option when compiling gtest
.
The following patch fixes the problem for me:
diff --git a/CMakeLists.txt b/CMakeLists.txt
index b4d8175..b53fbdd 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -220,7 +220,7 @@ CHECK_FUNCTION_EXISTS(_aligned_malloc HAVE__ALIGNED_MALLOC)
CHECK_C_COMPILER_FLAG(-fvisibility=hidden HAS_C_HIDDEN)
CHECK_CXX_COMPILER_FLAG(-fvisibility=hidden HAS_CXX_HIDDEN)
-if (RELEASE_BUILD)
+if (RELEASE_BUILD AND NOT BUILD_STATIC_AND_SHARED AND NOT BUILD_SHARED_LIBS)
if (HAS_C_HIDDEN)
set(EXTRA_C_FLAGS "${EXTRA_C_FLAGS} -fvisibility=hidden")
endif()
You may want to make more precise patch, which disables -fvisibility=hidden
for gtest
only, because other parts seem to work well with it.
The CMake script does require that ragel package is installed, but doesn't check if it is the case. It obviously fails when running make, without any obvious error message (recipe for target 'src/parser/Parser.cpp' failed).
I suggest adding an error case just like with Boost headers finding.
I think there is a bug regarding to UTF8 patterns. Here are my tests with Chinese strings:
Test 1:
Pattern: 测试
Scan: 这是一个测试
Compilation flags: HS_FLAG_UTF8
Result: Success
Test 2:
Pattern: \Q测试\E
Scan: 这是一个测试
Compilation flags: HS_FLAG_UTF8
Result: Failed
In Test 2, it should have passed, but instead it failed.
First, could I ask the question here?
Second, for input text "Host: www.google.com/mail", matched the rule "Host: (.*)/mail" , and I want to get the substring "www.google.com".
I think ptr + strlen("Host: ") is not a good idea, and I don't know how long for the target "www.google.com" in fact.
Thanks!
Compiling a database using hs_compile_multi() uses a lot of memory. I tried to compile 100,000 patterns on a system with 128G memory and it ran out of memory before finishing.
In a previous experience with Hyperscan (~2 years ago) it was able to handle that amount of patterns.
As reported in #5, there's a typo at line 78 of the root CMakeLists.txt : pacakges instead of packages.
When I compile my project with hyperscan 4.5.1, there is no error. But when I try to execute the binary, there is a critical error, same critical error when I compile samples/simplegrep.c
:
[root@05c8ec243f3d /home/yiwei.liyw/test/hyperscan-4.5.2/examples]
#gcc -I/home/deps/dest/usr/local/include/hs simplegrep.c /home/deps/dest/usr/local/lib64/libhs.a /usr/local/gcc-4.9.2/lib64/libstdc++.a -lm
[root@05c8ec243f3d /home/yiwei.liyw/test/hyperscan-4.5.2/examples]
#./a.out
./a.out: error while loading shared libraries: unexpected PLT reloc type 0x25
[root@05c8ec243f3d /home/yiwei.liyw/test/hyperscan-4.5.2/examples]
#ldd a.out
not a dynamic executable
Here is my environment:
[root@05c8ec243f3d /home/yiwei.liyw/test/hyperscan-4.5.2/examples]
#gcc --version
gcc (GCC) 4.9.2
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
[root@05c8ec243f3d /home/yiwei.liyw/test/hyperscan-4.5.2/examples]
#ld --version
GNU ld (GNU Binutils) 2.28
Copyright (C) 2017 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
[root@05c8ec243f3d /home/yiwei.liyw/test/hyperscan-4.5.2/examples]
#rpm -qf /lib64/libc.so.6
glibc-2.5-81.2
I think the problem is hyperscan use some relocation type that old glibc doesn't support. When I remove the static link to hyperscan from my project, the binary works well. So I'm pretty sure the problem is from hyperscan.
It would be awesome if hyperscan could be built using the Bazel build tool.
Would you like to add more error handling for return values from functions like the following?
simplegrep.c has the following in its comments:
Build instructions:
gcc -o simplegrep simplegrep.c $(pkg-config --cflags --libs libhs)
This command results in linking errors for me:
$ gcc -o simplegrep simplegrep.c $(pkg-config --cflags --libs libhs)
/usr/local/lib/libhs.a(ng_anchored_dots.cpp.o): In function `std::_Rb_tree_const_iterator<void*>::operator++()':
/usr/include/c++/4.8/bits/stl_tree.h:270: undefined reference to `std::_Rb_tree_increment(std::_Rb_tree_node_base const*)'
/usr/include/c++/4.8/bits/stl_tree.h:270: undefined reference to `std::_Rb_tree_increment(std::_Rb_tree_node_base const*)'
/usr/include/c++/4.8/bits/stl_tree.h:270: undefined reference to `std::_Rb_tree_increment(std::_Rb_tree_node_base const*)'
/usr/include/c++/4.8/bits/stl_tree.h:270: undefined reference to `std::_Rb_tree_increment(std::_Rb_tree_node_base const*)'
/usr/local/lib/libhs.a(ng_anchored_dots.cpp.o): In function `deallocate':
/usr/include/c++/4.8/ext/new_allocator.h:110: undefined reference to `operator delete(void*)'
...
27901 lines omitted
...
/usr/local/lib/libhs.a(engine_description.cpp.o):(.rodata._ZTVN3ue217EngineDescriptionE[_ZTVN3ue217EngineDescriptionE]+0x20): undefined reference to `__cxa_pure_virtual'
collect2: error: ld returned 1 exit status
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.8.4-2ubuntu1~14.04.1' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.1)
The following command works:
$ gcc -o simplegrep simplegrep.c $(pkg-config --cflags --libs libhs) -lstdc++ -lm
-lstdc++
fixes undefined reference to C++ library functions;
-lm
fixes undefined reference to ceil@@GLIBC_2.2.5
.
-lstdc++ -lm
can be added to libhs.pc
at least on some platforms.
I have compiled hyperscan 4.2.0 with address sanitizer enabled with gcc 4.8.0. Compiler options: -fsanitize=address
.
$ ./bin/unit-hyperscan --gtest_filter=Hyperscan/BadPattern.Block/10
Unit test executabled failed.
Raw log: https://gist.github.com/fd1d9298ed40f27d8169f75d7d9bcefa
Log symbolized with asan_symbolize.py and c++filt:
https://gist.github.com/03cc15922e679ac53e7d8a0aca807e16
I'm compiling hyperscan on CentOS 6.2, and compie error with following message:
/tmp/ccnZ8IHz.s: Assembler messages:
/tmp/ccnZ8IHz.s:671: Error: no such instruction: `vpbroadcastb %xmm3,%xmm3'
/tmp/ccnZ8IHz.s:680: Error: no such instruction: `vpbroadcastb %xmm2,%xmm2'
/tmp/ccnZ8IHz.s:855: Error: no such instruction: `vpbroadcastb %xmm3,%xmm3'
/tmp/ccnZ8IHz.s:864: Error: no such instruction: `vpbroadcastb %xmm2,%xmm2'
/tmp/ccnZ8IHz.s:1060: Error: no such instruction: `vpbroadcastb %xmm1,%xmm1'
/tmp/ccnZ8IHz.s:1178: Error: no such instruction: `vpbroadcastb %xmm1,%xmm1'
/tmp/ccnZ8IHz.s:1954: Error: no such instruction: `shlx %r8,%rax,%rax'
/tmp/ccnZ8IHz.s:2047: Error: no such instruction: `shlx %rdx,%rbx,%rdx'
/tmp/ccnZ8IHz.s:2055: Error: no such instruction: `shrx %r9d,%edx,%eax'
........
........
make[5]: *** [CMakeFiles/hs_exec_avx2.dir/src/runtime.c.o] Error 1
make[4]: *** [CMakeFiles/hs_exec_avx2.dir/all] Error 2
make[3]: *** [all] Error 2
I'm not sure this error is like #44 , But there is no fix or walk around in that issue. So I create new issue here.
My GCC version is: gcc version 5.2.0 (GCC)
, as version is: $as -v GNU assembler version 2.20.51.0.2 (x86_64-redhat-linux) using BFD version version 2.20.51.0.2-5.28.2.alios6 20091009
Hi Mdb256,
I read the following artical: http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/chimera-matching-engine-paper.pdf. This match engine supports full support for Capturing Groups and Full Pattern Support, which will resolve many issue I encounter today.
Is there any plan that you will port these feature from Chimera to hyperscan.
Will Chimera be open source project in the future.
-Tidy
clang version 4.0.1-svn305187-1~exp1 (branches/release_40)
boost 1.58.0+dfsg-5ubuntu3.1
hyperscan 4.5.2
Ubuntu xenial
/home/fgsch/hyperscan~/unit/internal/graph.cpp:208:5: error: unused typedef 'boost_concept_check208' [-Werror,-Wunused-local-typedef]
BOOST_CONCEPT_ASSERT((GraphConcept<SimpleG>));
^
/usr/include/boost/concept/assert.hpp:43:5: note: expanded from macro 'BOOST_CONCEPT_ASSERT'
BOOST_CONCEPT_ASSERT_FN(void(*)ModelInParens)
^
/usr/include/boost/concept/detail/general.hpp:79:7: note: expanded from macro 'BOOST_CONCEPT_ASSERT_FN'
BOOST_PP_CAT(boost_concept_check,__LINE__) \
^
/usr/include/boost/preprocessor/cat.hpp:22:32: note: expanded from macro 'BOOST_PP_CAT'
# define BOOST_PP_CAT(a, b) BOOST_PP_CAT_I(a, b)
^
/usr/include/boost/preprocessor/cat.hpp:29:34: note: expanded from macro 'BOOST_PP_CAT_I'
# define BOOST_PP_CAT_I(a, b) a ## b
^
<scratch space>:70:1: note: expanded from here
boost_concept_check208
Hello.
I have a question about pcapscan bin.
Whenever trying to execute the pcapscan, the network signature of request packet can't be found in HTTP.
But, I can find the network signature of response packet in HTTP. Did I have a mistake to use this?
What's difference between libhs.so and libhs_runtime.so, and when will we use the latter?
I was building hyperscan with sqlite-3.6.20 and got a compile error:
[ 98%] Building CXX object tools/hsbench/CMakeFiles/hsbench.dir/data_corpus.cpp.o
/home/yiwei.liyw/hyperscan-4.4.1/tools/hsbench/data_corpus.cpp: In function ‘std::vector<DataBlock> readCorpus(const string&)’:
/home/yiwei.liyw/hyperscan-4.4.1/tools/hsbench/data_corpus.cpp:113:37: error: ‘sqlite3_errstr’ was not declared in this scope
<< sqlite3_errstr(status);
^
make[2]: *** [tools/hsbench/CMakeFiles/hsbench.dir/data_corpus.cpp.o] Error 1
make[1]: *** [tools/hsbench/CMakeFiles/hsbench.dir/all] Error 2
make: *** [all] Error 2
Then I look into the sqlite.h
and find there is no declaration of sqlite3_errstr
, and this function declaration is exists on the latest version of sqlite3(sqlite3-3.18.0). So maybe add a dependency to SQLite, which is missing in current document. Or just don't use this function.
The required version for cmake should be 2.8.11. The string(TIMESTAMP ...) directive was released in 2.8.11 but is not available in previous versions.
It could be nice to provide a workaround as well.
Hi, for me it looks like there is an issue in the function hs_expression_info()
.
I expect that the function hs_expression_info()
has the same behaviour as the compile functions such as hs_compile()
. I expect that all patterns checked successfully with the hs_expression_info()
function will also be successfully process by the hs_compile()
function using the same pattern and flags, but it doesn't.
See the code example below. With a pattern that matches empty buffers and the HS_FLAG_ALLOWEMPTY
flag NOT set, the two functions behave different. In that case hs_expression_info()
returns HS_SUCCESS
but hs_compile()
fails.
I tested it with the latest sources on GitHub with tag v4.4.1.
#include <stdio.h>
#include <unistd.h>
#include "hs.h"
int main(int argc,char* argv[])
{
unsigned int hs_flags;
hs_expr_info_t *hs_info = NULL;
hs_compile_error_t *hs_error = NULL;
hs_database_t *hs_db;
const char pattern[] = ".*";
hs_flags = HS_FLAG_SINGLEMATCH; // | HS_FLAG_ALLOWEMPTY;
hs_error_t hs_ret = hs_expression_info(pattern, hs_flags, &hs_info, &hs_error);
if (hs_ret != HS_SUCCESS)
{
printf("Error evaluating regex failed (%d): %s\n", hs_ret, hs_error->message);
hs_free_compile_error(hs_error);
return EXIT_FAILURE;
}
hs_free_compile_error(hs_error);
free(hs_info);
hs_ret = hs_compile(pattern, hs_flags, HS_MODE_BLOCK, NULL, &hs_db, &hs_error);
if (hs_ret != HS_SUCCESS)
{
printf("Error compiling regex failed (%d): %s\n", hs_ret, hs_error->message);
hs_free_compile_error(hs_error);
return EXIT_FAILURE;
}
hs_free_compile_error(hs_error);
hs_free_database(hs_db);
return EXIT_SUCCESS;
}
Hi all, had anyone encounter this problem like me below:
[root@XXXX hs_build]# cmake --build . /usr/bin/make64 MAC=64 Scanning dependencies of target ragel_Parser [ 0%] Generating src/parser/Parser.cpp [ 0%] Built target ragel_Parser Scanning dependencies of target hs_exec [ 0%] Building C object CMakeFiles/hs_exec.dir/src/alloc.c.o [ 1%] Building C object CMakeFiles/hs_exec.dir/src/runtime.c.o /tmp/ccoj8u4q.s: Assembler messages: /tmp/ccoj8u4q.s:460: Error: no such instruction:
vpbroadcastb %xmm2,%xmm2'
/tmp/ccoj8u4q.s:470: Error: no such instruction: vpbroadcastb %xmm3,%xmm3' /tmp/ccoj8u4q.s:836: Error: no such instruction:
vpbroadcastb %xmm2,%xmm2'
/tmp/ccoj8u4q.s:846: Error: no such instruction: vpbroadcastb %xmm3,%xmm3' /tmp/ccoj8u4q.s:1191: Error: no such instruction:
vpbroadcastb %xmm1,%xmm1'
/tmp/ccoj8u4q.s:1318: Error: no such instruction: vpbroadcastb %xmm1,%xmm1' /tmp/ccoj8u4q.s:1933: Error: no such instruction:
shlx %r8,%rax,%rax'
/tmp/ccoj8u4q.s:2033: Error: no such instruction: shlx %rdx,%r11,%rdx' /tmp/ccoj8u4q.s:2052: Error: no such instruction:
shrx %esi,%eax,%eax'
/tmp/ccoj8u4q.s:2132: Error: no such instruction: shlx %rax,%r12,%rax' /tmp/ccoj8u4q.s:4303: Error: no such instruction:
shlx %rdx,%r11,%rdx'
/tmp/ccoj8u4q.s:4322: Error: no such instruction: shrx %r8d,%eax,%eax' /tmp/ccoj8u4q.s:4367: Error: no such instruction:
shlx %rax,%r15,%rax'
/tmp/ccoj8u4q.s:4437: Error: no such instruction: shlx %rsi,%rax,%rax' /tmp/ccoj8u4q.s:5203: Error: no such instruction:
shlx %rsi,%rax,%rax'
/tmp/ccoj8u4q.s:5388: Error: no such instruction: shlx %rsi,%rax,%rax' /tmp/ccoj8u4q.s:5675: Error: no such instruction:
shlx %rdx,%rax,%rdx'
/tmp/ccoj8u4q.s:5694: Error: no such instruction: shrx %edi,%eax,%eax' /tmp/ccoj8u4q.s:5747: Error: no such instruction:
shlx %rax,%rsi,%rax'
/tmp/ccoj8u4q.s:5838: Error: no such instruction: shlx %rdx,%rax,%rdx' /tmp/ccoj8u4q.s:5857: Error: no such instruction:
shrx %r8d,%eax,%eax'
/tmp/ccoj8u4q.s:5910: Error: no such instruction: shlx %rax,%rsi,%rax' /tmp/ccoj8u4q.s:6527: Error: no such instruction:
shlx %rdx,%r11,%rdx'
/tmp/ccoj8u4q.s:6546: Error: no such instruction: shrx %r8d,%eax,%eax' /tmp/ccoj8u4q.s:6590: Error: no such instruction:
shlx %rax,%r14,%rax'
/tmp/ccoj8u4q.s:6660: Error: no such instruction: shlx %rsi,%rax,%rax' /tmp/ccoj8u4q.s:7455: Error: no such instruction:
shlx %rdx,%r11,%rdx'
/tmp/ccoj8u4q.s:7474: Error: no such instruction: shrx %r8d,%eax,%eax' /tmp/ccoj8u4q.s:7519: Error: no such instruction:
shlx %rax,%r15,%rax'
/tmp/ccoj8u4q.s:7589: Error: no such instruction: shlx %rsi,%rax,%rax' /tmp/ccoj8u4q.s:8398: Error: no such instruction:
shlx %rdi,%rax,%rax'
/tmp/ccoj8u4q.s:8976: Error: no such instruction: shlx %r8,%rax,%rax' /tmp/ccoj8u4q.s:9384: Error: no such instruction:
shlx %rdx,%rax,%rdx'
/tmp/ccoj8u4q.s:9403: Error: no such instruction: shrx %r8d,%eax,%eax' /tmp/ccoj8u4q.s:9456: Error: no such instruction:
shlx %rax,%rdi,%rax'
/tmp/ccoj8u4q.s:9547: Error: no such instruction: shlx %rdx,%rax,%rdx' /tmp/ccoj8u4q.s:9566: Error: no such instruction:
shrx %ecx,%eax,%eax'
/tmp/ccoj8u4q.s:9619: Error: no such instruction: shlx %rax,%r8,%rax' /tmp/ccoj8u4q.s:9977: Error: no such instruction:
shlx %rcx,%r15,%rcx'
/tmp/ccoj8u4q.s:9996: Error: no such instruction: shrx %r9d,%edx,%edx' /tmp/ccoj8u4q.s:10039: Error: no such instruction:
shlx %rdx,%r14,%rdx'
/tmp/ccoj8u4q.s:10107: Error: no such instruction: shlx %rsi,%rdx,%rdx' make64[2]: *** [CMakeFiles/hs_exec.dir/src/runtime.c.o] Error 1 make64[1]: *** [CMakeFiles/hs_exec.dir/all] Error 2 make64: *** [all] Error 2
It seems like a problem about cpu instruction set, my /proc/cpuinfo return :
processor : 11
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
stepping : 2
microcode : 0x31
cpu MHz : 2400.000
cache size : 15360 KB
physical id : 1
siblings : 6
core id : 5
cpu cores : 6
apicid : 26
initial apicid : 26
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 4793.25
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
[root@XXXX hs_build]# gcc -march=native -Q --help=target | grep march
-march= core-avx2
when I forbidden the instruction set avx and bmi2, then it could complete the compilation, but the hyperscan 's performance is just about 500Mb/s, it's really long distance with the official numbers. I doubt that it maybe relate with the instruction sets.
We I try to upgrade Hyperscan from v4.1.0 to v4.2.0 for homebrew, build failed at OSX mavericks with Clang: 6.0 build 600
/tmp/hyperscan-20160701-38604-nh3uxt/hyperscan-4.2.0/src/rose/rose_build_misc.cpp:883:16: error: chosen constructor is explicit in copy-initialization
return {};
^~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/set:428:14: note: constructor declared here
explicit set(const value_compare& __comp = value_compare())
^
Clang: 6.0 build 600
OS X: 10.9.5-x86_64
Xcode: 6.2
Please check the full build log at
http://bot.brew.sh/job/Homebrew%20Core%20Pull%20Requests/version=mavericks/4050/consoleFull
Does Hyperscan support gzip webpages ? I run a test on www.bing.com with ./pcapscan [pattern] [pcap file] and hyperscan Couldn't find 'Legal' or any pattern
It would be grate if you help me .
best regards
These 2 flags are conflicted .
When I want to Use these two flags together the flag HS_FLAG_SOM_LEFTMOST does not work correctly and at the eventHandler() function the value of from
is always 0
zero !!!
Can I call hs_alloc_scratch() on the same database in multiple threads to allocation thread local scratch space concurrently without guarding the function call by a mutex?
I am using version 4.5.1. In my current code, this is what I have done already. However, from time to time, some of the threads fails to search the database with error in validScratch(). Sometimes it gave error "bad magic 0x901343" , and sometimes it failed with the error "bad state size".
Is it possible that this problem was caused by the fact that I am allocating scratch space instances concurrently?
Some parameters (like "${PROJECT_BINARY_DIR}/bin" and "${CMAKE_CURRENT_SOURCE_DIR}/autogen.py") are passed to CMake commands in your build scripts without enclosing them by quotation marks. I see that these places will result in build difficulties if the contents of the used variables will contain special characters like semicolons.
I would recommend to apply advices from a wiki article.
The issue arose when packaging hyperscan for Fedora/EPEL:
"In lines 179-244, this CMakeList.txt implements a scheme which manipulates and overrides CFLAGS/CXXFLAGS to values which are incompatible to Fedora (== this package is being miscompiled). This alone qualifies this package as non-eligible for Fedora, unless this issue can be worked around"
Dear all,
I am a software engineer, I want to use hyperscan on arm platform , when I crosscompile the source code , I get the folling errors:
-- Looking for include file intrin.h - not found
......
CMake Error at cmake/arch.cmake:10 (message):
No intrinsics header found
Call Stack (most recent call first):
CMakeLists.txt:260 (include)
-- Configuring incomplete, errors occurred!
I search all files on my os , I could not find the intrin.h file.
Could you help me about this issue?
Thanks a lot.
Best Regards
These is the build informaton :
audit_admin@audit1:~/test/lib_src/hyperscan-4.2.0/build$ cmake -DBUILD_STATIC_AND_SHARED=1 ../
-- The C compiler identification is GNU 4.8.3
-- The CXX compiler identification is GNU 4.8.3
-- Check for working C compiler: /opt/arm-2014.05/bin/arm-none-linux-gnueabi-gcc
-- Check for working C compiler: /opt/arm-2014.05/bin/arm-none-linux-gnueabi-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /opt/arm-2014.05/bin/arm-none-linux-gnueabi-g++
-- Check for working CXX compiler: /opt/arm-2014.05/bin/arm-none-linux-gnueabi-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Performing Test ARCH_64_BIT
-- Performing Test ARCH_64_BIT - Failed
-- Performing Test ARCH_32_BIT
-- Performing Test ARCH_32_BIT - Failed
-- Default build type 'Release with debug info'
-- Boost version: 1.61.0
-- Found PythonInterp: /usr/bin/python (found version "2.7.6")
-- Building shared libraries
-- g++ version 4.8.3
-- Building for current host CPU
-- Looking for include file unistd.h
-- Looking for include file unistd.h - found
-- Looking for include file intrin.h
-- Looking for include file intrin.h - not found
-- Looking for C++ include intrin.h
-- Looking for C++ include intrin.h - not found
-- Looking for include file tmmintrin.h
-- Looking for include file tmmintrin.h - not found
-- Looking for include file x86intrin.h
-- Looking for include file x86intrin.h - not found
-- Looking for C++ include x86intrin.h
-- Looking for C++ include x86intrin.h - not found
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Looking for _aligned_malloc
-- Looking for _aligned_malloc - not found
-- Performing Test HAS_C_HIDDEN
-- Performing Test HAS_C_HIDDEN - Success
-- Performing Test HAS_CXX_HIDDEN
-- Performing Test HAS_CXX_HIDDEN - Success
CMake Error at cmake/arch.cmake:10 (message):
No intrinsics header found
Call Stack (most recent call first):
CMakeLists.txt:260 (include)
-- Configuring incomplete, errors occurred!
Hi there, can you point me to the VS solution mentioned on your page? I don't find those files in the repo. Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.