mozilla-services / hindsight Goto Github PK

Hindsight - light weight data processing skeleton

License: Mozilla Public License 2.0

CMake 2.28% C 89.36% Lua 7.09% Shell 0.12% Dockerfile 1.15%

hindsight's Introduction

Hindsight

Overview

Hindsight is a C based data processing infrastructure based on the lua sandbox project. I have received several inquiries about a lighter weight and faster data pipeline with delivery guarantees to replace Heka. Hindsight is that light weight skeleton around the same lua sandbox offering 'at least once' delivery semantics. The skeleton is supplemented by extension packages including hundreds of data structures, algorithms, plugins, parsers and grammars. The extensions repository is where most of the active development is happening now as the core infrastructure (Hindsight and the Lua Sandbox) is stable and changes infrequently. There is also a Hindsight Administration UI available for monitoring, debugging and plugin management (you can check out a running instance here: hsadmin)

Full Documentation
Support
- Chat: Matrix
- Mailing list: https://mail.mozilla.org/listinfo/hindsight

Build

Prerequisites

Clang 3.1 or GCC 4.7+
CMake (3.6+) - http://cmake.org/cmake/resources/software.html
lua_sandbox (1.2.3+) - https://github.com/mozilla-services/lua_sandbox
OpenSSL (1.0.x+, optional)

CMake Build Instructions

git clone https://github.com/mozilla-services/hindsight.git
cd hindsight
mkdir release
cd release

# Linux
cmake -DCMAKE_BUILD_TYPE=release ..
make
ctest
cpack -G TGZ # (DEB|RPM|ZIP)

# Cross platform support is planned but not supported yet

By default hindsight is linked against OpenSSL and configured to set locking callbacks in the library to ensure proper threaded operation. If this functionality is not desired the cmake build option -DWITHOUT_OPENSSL=true can be used to disable this, for example if you are not using any sandboxes/modules that make use of OpenSSL and do not want the dependency.

Releases

The main branch is the current release and is considered stable at all times.
New versions can be released as frequently as every two weeks (our sprint cycle). The only exception would be for a high priority patch.
All active work is flagged with the sprint milestone and tracked in the project dashboard.
New releases occur the day after the sprint finishes.
- The version in the dev branch is updated
- The changes are merged into main
- A new tag is created

Docker Images

Docker images are constructed from the main and dev branches and can be pulled, or built using the Dockerfile.

Note that the Docker image built here is only a bare bones image containing just lua_sandbox and hindsight. For a more full featured image that also contains all of the extensions, see the Docker image for the extensions repo.

Contributions

All pull requests must be made against the dev branch, direct commits to main are not permitted.
All non trivial contributions should start with an issue being filed (if it is a new feature please propose your design/approach before doing any work as not all feature requests are accepted).

hindsight's People

Contributors

Stargazers

Watchers

hindsight's Issues

Move the sandbox state preservation files under the output_path

Allow configuration variables to use/specify tables

Add a backpressure setting that is based on remaining disk space

e.g. if there are less than N * output_size bytes available on the output_path volume apply back pressure

analysis sandbox inject_message: Can't set message Hostname

I know this is the expected behavior as described in the documentation: https://github.com/mozilla-services/lua_sandbox/blob/master/docs/heka/analysis.md#inject_message but what is the reasoning behind preventing the sandbox to override the message Hostname?

Here is my use case: I'm using an analysis sandbox to decorate messages going through the pipeline with new fields or reformat existing fields, so that these messages can then be forwarded to different output sandboxes but I do want to retain the original Hostname value of these messages.

So here is a simplified example of what I want to do in my analysis sandbox:

function process_message()
    local msg = decode_message(read_message("raw"))
    -- [ add/remove/update some fields in msg.Fields ]
    inject_message(msg) -- the new injected message should keep the original msg.Hostname value
    return 0
end

function timer_event(ns)
end

Would it be possible to allow Hostname override in analysis sandbox inject_message?

improve out of disk space handling

i.e, at least preserve the last checkpoint successfully written to disk for debug purposes

implement the new inject_message_callback return values

mozilla-services/lua_sandbox#184

Expand the input buffer more than the needed size

Prevent a lot of re-allocations when the input messages are growing by small amounts

Improve process_message sampling

On low volume plugins samples rarely occurs (message matcher and process message sampling have to be independent). Analysis plugins each need their own sampling flag for process_message

Set remove_entries in a sandbox configuration

I would like to set remove_entries in my sandbox' configuration file. It seems to me that this is not currently supported by Hindsight.

Would this addition be considered?

Need a way for output plugins to handle async delivery checkpointing

Batch mode checkpointing is all or none. There needs to be a way to set a checkpoint with a sequence number (available to process_message and timer_event).

Race condition hang when using popen from multiple sandboxes at once

Blocks on __lll_lock_wait_private

hindsight is doing too much disk noise ;-)

Every second, hindsight is creating hindsight.tsv.tmp, filling it, then moving it to hindsight.cp.

If nothing change, this do unnecessary disk writes. This consumes resources and energy.

Maybe keep the previous cp in memory and rename only if changed?

Add configuration options to throttle inject_message

The throttle rate should independently apply to process_message and timer_event; even allowing it to be fully disabled.

Investigate poor performance in the multithreaded analysis configuration

Compilation errors on raspberry pi

https://mail.mozilla.org/pipermail/heka/2015-July/000655.html

pruning log files when some sandboxes are stopped

When sandboxes are stopped either because of a failures or because we manually stop these sandboxes by creating .off files in the related load directory, the lines related to the stopped sandboxes in the hindsight.cp checkpoint file are not purged, and then the prune_input.lua input sandbox cannot prune the hindsight log files anymore.
Is it an expected behaviour?
How would you manage this use case of purging the log files when some sandboxes have been stopped?

Allow slow running plugins to be stopped

If a plugin is causing the system to back pressure it would we useful to have the ability to automatically stop it.

Allow output plugins to subscribe to a particular queue

read_queue = "(input|analysis|both)" -- default to both, the current behavior

Thread selection for analysis plugin deployment should be based on utilization

If a specific thread is not specified in the configuration the analysis plugin should be deployed to the thread with the lowest utilization. See: mozilla-services/hindsight_admin#10.

Note: Running plugins that are restarted must/will remain on their existing thread.

allow to force Hindsight to shutdown when an essential sandbox dies

I need something equivalent to the can_exit config in Heka.

For example, if the main sandbox of my Hindsight setup dies (for example because of a permission issue, or because my disk is full), I would like Hindsight to shutdown.
Does Hindsight support this use case?

Ideally Hindsight could try to restart the failing sandbox a configurable number of times, and if the sandbox is still dying, Hindsight would completely shutdown.

What do you think?

ctest fails: test_config (SEGFAULT)

When I run ctest after Hindsight has been successfully compiled on ARM platform (raspberry pi), I get the segfault error below:

$ ctest -VV --debug

/cmake-3.6.1/Source/cmCTest.cxx:435 Here: 435

/cmake-3.6.1/Source/cmCTest.cxx:687 UpdateCTestConfiguration  from :/hindsight/release/DartConfiguration.tcl

/cmake-3.6.1/Source/cmCTest.cxx:697 Parse Config file:/hindsight/release/DartConfiguration.tcl

/cmake-3.6.1/Source/cmCTest.cxx:447 Here: 447

/cmake-3.6.1/Source/cmCTest.cxx:2223 * Read custom CTest configuration directory: /hindsight/release

/cmake-3.6.1/Source/cmCTest.cxx:2227 * Check for file: /hindsight/release/CTestCustom.cmake

/cmake-3.6.1/Source/cmCTest.cxx:2247 * Check for file: /hindsight/release/CTestCustom.ctest

/cmake-3.6.1/Source/cmCTest.cxx:687 UpdateCTestConfiguration  from :/hindsight/release/DartConfiguration.tcl

/cmake-3.6.1/Source/cmCTest.cxx:697 Parse Config file:/hindsight/release/DartConfiguration.tcl

/cmake-3.6.1/Source/CTest/cmCTestTestHandler.cxx:461 Test project /hindsight/release

/cmake-3.6.1/Source/CTest/cmCTestTestHandler.cxx:1348 Constructing a list of tests

/cmake-3.6.1/Source/CTest/cmCTestTestHandler.cxx:1926 Add test: test_config

/cmake-3.6.1/Source/CTest/cmCTestTestHandler.cxx:1970 Set test directory: /hindsight/release/src/test

/cmake-3.6.1/Source/CTest/cmCTestTestHandler.cxx:1398 Done constructing a list of tests

/cmake-3.6.1/Source/cmCTest.cxx:146    Current_Time: Sep 12 13:17 UTC

/cmake-3.6.1/Source/cmCTest.cxx:146    Current_Time: Sep 12 13:17 UTC

/cmake-3.6.1/Source/CTest/cmCTestMultiProcessHandler.cxx:798 Checking test dependency graph...

/cmake-3.6.1/Source/CTest/cmCTestMultiProcessHandler.cxx:830 Checking test dependency graph end

/cmake-3.6.1/Source/CTest/cmCTestMultiProcessHandler.cxx:121 test 1

/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:416     Start 1: test_config

/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:537
1: Test command: /hindsight/release/src/test/test_config

/cmake-3.6.1/Source/cmCTest.cxx:146    Current_Time: Sep 12 13:17 UTC

/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:657 1: Test timeout computed to be: 1500

/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:60 1: 1473686241625913873 [error] config_parser loading cfg/extra.cfg failed: invalid option: 'extra'

/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:682 1/
/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:684 1
/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:700 Test
/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:707  #1:
/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:708
/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:740 test_config ......................
/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:742 Testing test_config ...
/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:230 ***Exception:
/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:233 SegFault
/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:260   0.01 sec

/cmake-3.6.1/Source/CTest/cmCTestRunTest.cxx:263 1473686241625913873 [error] config_parser loading cfg/extra.cfg failed: invalid option: 'extra'


/cmake-3.6.1/Source/cmCTest.cxx:146    Current_Time: Sep 12 13:17 UTC

/cmake-3.6.1/Source/cmCTest.cxx:146    Current_Time: Sep 12 13:17 UTC

/cmake-3.6.1/Source/cmCTest.cxx:146    Current_Time: Sep 12 13:17 UTC

/cmake-3.6.1/Source/CTest/cmCTestTestHandler.cxx:510
0% tests passed, 1 tests failed out of 1

/cmake-3.6.1/Source/CTest/cmCTestTestHandler.cxx:518
Total Test time (real) =   0.03 sec

/cmake-3.6.1/Source/CTest/cmCTestTestHandler.cxx:523
The following tests FAILED:

/cmake-3.6.1/Source/CTest/cmCTestTestHandler.cxx:539      1 - test_config (SEGFAULT)

/cmake-3.6.1/Source/cmCTest.cxx:977 Errors while running CTest

/cmake-3.6.1/Source/cmCTest.cxx:2195 Running a test(s) failed returning : 8

Then I tried to manually run the test_config program which has produced a core dumped that I can provide if needed:

$ ../../release/src/test/test_config 
1473686868112134801 [error] config_parser loading cfg/extra.cfg failed: invalid option: 'extra'
Segmentation fault (core dumped)

$ gdb ../../release/src/test/test_config core 
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /hindsight/release/src/test/test_config...(no debugging symbols found)...done.
[New LWP 562]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `../../release/src/test/test_config'.
Program terminated with signal 11, Segmentation fault.
#0  0x76f348e0 in lua_close () from /usr/local/lib/libluasandbox.so.0
(gdb) bt
#0  0x76f348e0 in lua_close () from /usr/local/lib/libluasandbox.so.0
#1  0x0000b8ac in hs_free_checkpoint_reader ()
#2  0x0000d5fc in all_tests ()
#3  0x00009fdc in main ()
(gdb)

Note that the commit hash at which Hindsight was compiled is: 4f5dcdc

Add a hs-cat program (equivalent to heka-cat)

In mozilla-services/heka#1329 (comment) you said you were experimenting with a new hs-cat program which was 6x faster than heka-cat.

Investigate input plugin threads not being shutdown cleanly

join timed out, cancelling the thread appears to be happening more than it should

Clear steps for setting up hindsight

I have read the documentation provided here and on lua_sandbox(_extensions). My documentation-fu is failing to help me find clear steps on setting up dependencies and how to install hindsight properly.
I have built:
lua_sandbox
hindsight
librdkafka

I have attempted to build lua_sandbox_extensions but it fails due to not having GEOIP_LIBRARY defined. Is there an easy way to build the individual packages without building everything? Also are there any plans to add documentation on how to install and run hindsight?

Thank you in advance for your time

Tearing down blocked inputs can segfault

If an input plugin is blocked on a system call and is orphaned and then becomes unblocked during the process shutdown there is a race condition where it can segfault while exiting.

Compilation fails with `LUASANDBOX_INCLUDE_DIR does not exist`

Thanks for developing hindsight - it looks like it'll be a great successor to Heka :-)

I'm trying to compile hindsight on (essentially) an Ubuntu Trusty machine, and it fails with

CMake Error at /tmp/app/lua_sandbox/release/luasandboxConfig.cmake:13 (message):
  File or directory /tmp/include referenced by variable
  LUASANDBOX_INCLUDE_DIR does not exist !
Call Stack (most recent call first):
  /tmp/app/lua_sandbox/release/luasandboxConfig.cmake:29 (set_and_check)
  CMakeLists.txt:22 (find_package)

There a a couple of prior dependency problems which I /believe/ I've solved, but I'll include their preamble in case I've screwed them up.

I don't know enough CMake to fix the LUASANDBOX_INCLUDE_DIR reference but, to be honest, I think I've followed the build instructions to the letter, so this might be a problem deserving a fix outside my local scope.

Ubuntu Trusty needs CMake to be upgraded, and doesn't have the lua_sandbox. Here's the script I'm running to install all of this, followed by a run of script. It's (naively!) based on the compilation instructions in the hindsight and lua_sandbox repos. (The script has a set -x, so its commands are intermingled with its output.)

#!/usr/bin/env bash
set -x
BPDIR=$1
############## CMAKE UPGRADE ####################
cd $BPDIR
curl --location -s S3://cmake-3.6.2-Linux-x86_64.tar.gz \
| tar xfz -
export PATH=$BPDIR/cmake-3.6.2-Linux-x86_64/bin:$PATH

############## LUASANDBOX #######################
cd $BPDIR
curl --location -s https://github.com/mozilla-services/lua_sandbox/archive/v1.2.0.tar.gz \
| tar xfz -
mv lua_sandbox-1.2.0 lua_sandbox
mkdir -p lua_sandbox/release
cd lua_sandbox/release
cmake -DCMAKE_BUILD_TYPE=release ..
make
ctest
cpack -G TGZ
export luasandbox_DIR=$BPDIR/lua_sandbox/release

############## HINDSIGHT ########################
cd $BPDIR
git clone https://github.com/mozilla-services/hindsight.git
mkdir -p hindsight/release
cd hindsight/release
cmake -DCMAKE_BUILD_TYPE=release ..
make
ctest
cpack -G TGZ

Here's the full output:

+ cd /tmp/app
+ curl --location -s S3://cmake-3.6.2-Linux-x86_64.tar.gz
+ tar xfz -
+ export PATH=/tmp/app/cmake-3.6.2-Linux-x86_64/bin:/usr/local/bin:/usr/bin:/bin
+ PATH=/tmp/app/cmake-3.6.2-Linux-x86_64/bin:/usr/local/bin:/usr/bin:/bin
+ cd /tmp/app
+ curl --location -s https://github.com/mozilla-services/lua_sandbox/archive/v1.2.0.tar.gz
+ tar xfz -
+ mv lua_sandbox-1.2.0 lua_sandbox
+ mkdir -p lua_sandbox/release
+ cd lua_sandbox/release
+ cmake -DCMAKE_BUILD_TYPE=release ..
-- The C compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
The optional documentation tools were not found; the doc target has not been created
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/app/lua_sandbox/release
+ make
Scanning dependencies of target luasandboxutil
[  1%] Building C object src/util/CMakeFiles/luasandboxutil.dir/heka_message.c.o
[  2%] Building C object src/util/CMakeFiles/luasandboxutil.dir/heka_message_matcher.c.o
[  4%] Building C object src/util/CMakeFiles/luasandboxutil.dir/heka_message_matcher_parser.c.o
[  5%] Building C object src/util/CMakeFiles/luasandboxutil.dir/input_buffer.c.o
[  6%] Building C object src/util/CMakeFiles/luasandboxutil.dir/output_buffer.c.o
[  8%] Building C object src/util/CMakeFiles/luasandboxutil.dir/protobuf.c.o
[  9%] Building C object src/util/CMakeFiles/luasandboxutil.dir/running_stats.c.o
[ 10%] Building C object src/util/CMakeFiles/luasandboxutil.dir/string.c.o
[ 12%] Building C object src/util/CMakeFiles/luasandboxutil.dir/string_matcher.c.o
[ 13%] Building C object src/util/CMakeFiles/luasandboxutil.dir/util.c.o
[ 14%] Linking C shared library libluasandboxutil.so
[ 14%] Built target luasandboxutil
Scanning dependencies of target luasandbox
[ 16%] Building C object src/CMakeFiles/luasandbox.dir/luasandbox.c.o
[ 17%] Building C object src/CMakeFiles/luasandbox.dir/luasandbox_output.c.o
[ 18%] Building C object src/CMakeFiles/luasandbox.dir/luasandbox_serialize.c.o
[ 20%] Building C object src/CMakeFiles/luasandbox.dir/lua/lapi.c.o
[ 21%] Building C object src/CMakeFiles/luasandbox.dir/lua/lauxlib.c.o
[ 22%] Building C object src/CMakeFiles/luasandbox.dir/lua/lbaselib.c.o
[ 24%] Building C object src/CMakeFiles/luasandbox.dir/lua/lcode.c.o
[ 25%] Building C object src/CMakeFiles/luasandbox.dir/lua/ldblib.c.o
[ 26%] Building C object src/CMakeFiles/luasandbox.dir/lua/ldebug.c.o
[ 28%] Building C object src/CMakeFiles/luasandbox.dir/lua/ldo.c.o
[ 29%] Building C object src/CMakeFiles/luasandbox.dir/lua/ldump.c.o
[ 30%] Building C object src/CMakeFiles/luasandbox.dir/lua/lfunc.c.o
[ 32%] Building C object src/CMakeFiles/luasandbox.dir/lua/lgc.c.o
[ 33%] Building C object src/CMakeFiles/luasandbox.dir/lua/linit.c.o
[ 34%] Building C object src/CMakeFiles/luasandbox.dir/lua/liolib.c.o
[ 36%] Building C object src/CMakeFiles/luasandbox.dir/lua/llex.c.o
[ 37%] Building C object src/CMakeFiles/luasandbox.dir/lua/lmathlib.c.o
[ 38%] Building C object src/CMakeFiles/luasandbox.dir/lua/lmem.c.o
[ 40%] Building C object src/CMakeFiles/luasandbox.dir/lua/loadlib.c.o
[ 41%] Building C object src/CMakeFiles/luasandbox.dir/lua/lobject.c.o
[ 42%] Building C object src/CMakeFiles/luasandbox.dir/lua/lopcodes.c.o
[ 44%] Building C object src/CMakeFiles/luasandbox.dir/lua/loslib.c.o
[ 45%] Building C object src/CMakeFiles/luasandbox.dir/lua/lparser.c.o
[ 46%] Building C object src/CMakeFiles/luasandbox.dir/lua/lstate.c.o
[ 48%] Building C object src/CMakeFiles/luasandbox.dir/lua/lstring.c.o
[ 49%] Building C object src/CMakeFiles/luasandbox.dir/lua/lstrlib.c.o
[ 50%] Building C object src/CMakeFiles/luasandbox.dir/lua/ltable.c.o
[ 52%] Building C object src/CMakeFiles/luasandbox.dir/lua/ltablib.c.o
[ 53%] Building C object src/CMakeFiles/luasandbox.dir/lua/ltm.c.o
[ 54%] Building C object src/CMakeFiles/luasandbox.dir/lua/lundump.c.o
[ 56%] Building C object src/CMakeFiles/luasandbox.dir/lua/lvm.c.o
[ 57%] Building C object src/CMakeFiles/luasandbox.dir/lua/lzio.c.o
[ 58%] Linking C shared library libluasandbox.so
[ 58%] Built target luasandbox
Scanning dependencies of target test_input_buffer
[ 60%] Building C object src/util/test/CMakeFiles/test_input_buffer.dir/test_input_buffer.c.o
[ 61%] Linking C executable test_input_buffer
[ 61%] Built target test_input_buffer
Scanning dependencies of target test_output_buffer
[ 62%] Building C object src/util/test/CMakeFiles/test_output_buffer.dir/test_output_buffer.c.o
[ 64%] Linking C executable test_output_buffer
[ 64%] Built target test_output_buffer
Scanning dependencies of target test_string_matcher
[ 65%] Building C object src/util/test/CMakeFiles/test_string_matcher.dir/test_string_matcher.c.o
[ 66%] Linking C executable test_string_matcher
[ 66%] Built target test_string_matcher
Scanning dependencies of target test_heka_message
[ 68%] Building C object src/util/test/CMakeFiles/test_heka_message.dir/test_heka_message.c.o
[ 69%] Linking C executable test_heka_message
[ 69%] Built target test_heka_message
Scanning dependencies of target test_string
[ 70%] Building C object src/util/test/CMakeFiles/test_string.dir/test_string.c.o
[ 72%] Linking C executable test_string
[ 72%] Built target test_string
Scanning dependencies of target test_running_stats
[ 73%] Building C object src/util/test/CMakeFiles/test_running_stats.dir/test_running_stats.c.o
[ 74%] Linking C executable test_running_stats
[ 74%] Built target test_running_stats
Scanning dependencies of target test_protobuf
[ 76%] Building C object src/util/test/CMakeFiles/test_protobuf.dir/test_protobuf.c.o
[ 77%] Linking C executable test_protobuf
[ 77%] Built target test_protobuf
Scanning dependencies of target test_heka_message_matcher
[ 78%] Building C object src/util/test/CMakeFiles/test_heka_message_matcher.dir/test_heka_message_matcher.c.o
[ 80%] Linking C executable test_heka_message_matcher
[ 80%] Built target test_heka_message_matcher
Scanning dependencies of target test_util
[ 81%] Building C object src/util/test/CMakeFiles/test_util.dir/test_util.c.o
[ 82%] Linking C executable test_util
[ 82%] Built target test_util
Scanning dependencies of target luasandboxtest
[ 84%] Building C object src/test/CMakeFiles/luasandboxtest.dir/sandbox.c.o
[ 85%] Linking C shared library libluasandboxtest.so
[ 85%] Built target luasandboxtest
Scanning dependencies of target luasandboxheka
[ 86%] Building C object src/heka/CMakeFiles/luasandboxheka.dir/message.c.o
[ 88%] Building C object src/heka/CMakeFiles/luasandboxheka.dir/read_message_zc.c.o
[ 89%] Building C object src/heka/CMakeFiles/luasandboxheka.dir/sandbox.c.o
[ 90%] Building C object src/heka/CMakeFiles/luasandboxheka.dir/stream_reader.c.o
[ 92%] Linking C shared library libluasandboxheka.so
[ 92%] Built target luasandboxheka
Scanning dependencies of target test_heka_sandbox
[ 93%] Building C object src/heka/test/CMakeFiles/test_heka_sandbox.dir/test_heka_sandbox.c.o
[ 94%] Linking C executable test_heka_sandbox
[ 94%] Built target test_heka_sandbox
Scanning dependencies of target lsb_heka_cat
[ 96%] Building C object src/cli/CMakeFiles/lsb_heka_cat.dir/lsb_heka_cat.c.o
[ 97%] Linking C executable lsb_heka_cat
[ 97%] Built target lsb_heka_cat
Scanning dependencies of target test_generic_sandbox
[ 98%] Building C object src/test/CMakeFiles/test_generic_sandbox.dir/test_generic_sandbox.c.o
[100%] Linking C executable test_generic_sandbox
[100%] Built target test_generic_sandbox
+ ctest
Test project /tmp/app/lua_sandbox/release
      Start  1: test_input_buffer
 1/14 Test  #1: test_input_buffer ..................   Passed    0.00 sec
      Start  2: test_output_buffer
 2/14 Test  #2: test_output_buffer .................   Passed    0.00 sec
      Start  3: test_protobuf
 3/14 Test  #3: test_protobuf ......................   Passed    0.00 sec
      Start  4: test_string_matcher
 4/14 Test  #4: test_string_matcher ................   Passed    0.00 sec
      Start  5: test_running_stats
 5/14 Test  #5: test_running_stats .................   Passed    0.00 sec
      Start  6: test_util
      Start  7: test_heka_message
 6/14 Test  #6: test_util ..........................   Passed    0.15 sec
 7/14 Test  #7: test_heka_message ..................   Passed    0.00 sec
      Start  8: test_heka_message_matcher
 8/14 Test  #8: test_heka_message_matcher ..........   Passed    0.61 sec
      Start  9: test_string
 9/14 Test  #9: test_string ........................   Passed    0.00 sec
      Start 10: test_move_heka_sandbox_tests
10/14 Test #10: test_move_heka_sandbox_tests .......   Passed    0.00 sec
      Start 11: test_heka_sandbox
11/14 Test #11: test_heka_sandbox ..................***Failed    0.02 sec
      Start 12: test_move_luasandbox_lua_modules
12/14 Test #12: test_move_luasandbox_lua_modules ...   Passed    0.00 sec
      Start 13: test_move_luasandbox_tests
13/14 Test #13: test_move_luasandbox_tests .........   Passed    0.00 sec
      Start 14: test_generic_sandbox
14/14 Test #14: test_generic_sandbox ...............   Passed    3.60 sec
93% tests passed, 1 tests failed out of 14
Total Test time (real) =   4.40 sec
The following tests FAILED:
         11 - test_heka_sandbox (Failed)
Errors while running CTest
+ cpack -G TGZ
CPack: Create package using TGZ
CPack: Install projects
CPack: - Run preinstall target for: luasandbox
CPack: - Install project: luasandbox
CPack: Create package
CPack: - package: /tmp/app/lua_sandbox/release/luasandbox-1.2.0-Linux.tar.gz generated.
+ export luasandbox_DIR=/tmp/app/lua_sandbox/release
+ luasandbox_DIR=/tmp/app/lua_sandbox/release
+ cd /tmp/app
+ git clone https://github.com/mozilla-services/hindsight.git
Cloning into 'hindsight'...
+ mkdir -p hindsight/release
+ cd hindsight/release
+ cmake -DCMAKE_BUILD_TYPE=release ..
-- The C compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
CMake Error at /tmp/app/lua_sandbox/release/luasandboxConfig.cmake:13 (message):
  File or directory /tmp/include referenced by variable
  LUASANDBOX_INCLUDE_DIR does not exist !
Call Stack (most recent call first):
  /tmp/app/lua_sandbox/release/luasandboxConfig.cmake:29 (set_and_check)
  CMakeLists.txt:22 (find_package)
-- Configuring incomplete, errors occurred!
See also "/tmp/app/hindsight/release/CMakeFiles/CMakeOutput.log".
+ make
make: *** No targets specified and no makefile found.  Stop.
+ ctest
*********************************
*********************************
No test configuration file found!
Usage
  ctest [options]
+ cpack -G TGZ
CPack: Create package using TGZ
CPack: Install projects
CPack: - Run preinstall target for: hindsight
CPack Error: Problem running install command: /tmp/app/cmake-3.6.2-Linux-x86_64/bin/cmake --build . --target "preinstall"
Please check /tmp/app/hindsight/release/_CPack_Packages/Linux/TGZ/PreinstallOutput.log for errors

As per the two mentions in the output, here are the 2 log files referenced:

CMakeOutput.log:

The system is: Linux - 3.19.0-68-generic - x86_64
Compiling the C compiler identification source file "CMakeCCompilerId.c" succeeded.
Compiler: /usr/bin/cc 
Build flags: 
Id flags: 

The output was:
0


Compilation of the C compiler identification source "CMakeCCompilerId.c" produced "a.out"

The C compiler identification is GNU, found in "/tmp/app/hindsight/release/CMakeFiles/3.6.2/CompilerIdC/a.out"

Determining if the C compiler works passed with the following output:
Change Dir: /tmp/app/hindsight/release/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_fa332/fast"
/usr/bin/make -f CMakeFiles/cmTC_fa332.dir/build.make CMakeFiles/cmTC_fa332.dir/build
make[1]: Entering directory `/tmp/app/hindsight/release/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_fa332.dir/testCCompiler.c.o
/usr/bin/cc     -o CMakeFiles/cmTC_fa332.dir/testCCompiler.c.o   -c /tmp/app/hindsight/release/CMakeFiles/CMakeTmp/testCCompiler.c
Linking C executable cmTC_fa332
/tmp/app/cmake-3.6.2-Linux-x86_64/bin/cmake -E cmake_link_script CMakeFiles/cmTC_fa332.dir/link.txt --verbose=1
/usr/bin/cc       CMakeFiles/cmTC_fa332.dir/testCCompiler.c.o  -o cmTC_fa332 -rdynamic 
make[1]: Leaving directory `/tmp/app/hindsight/release/CMakeFiles/CMakeTmp'


Detecting C compiler ABI info compiled with the following output:
Change Dir: /tmp/app/hindsight/release/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_ab41b/fast"
/usr/bin/make -f CMakeFiles/cmTC_ab41b.dir/build.make CMakeFiles/cmTC_ab41b.dir/build
make[1]: Entering directory `/tmp/app/hindsight/release/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_ab41b.dir/CMakeCCompilerABI.c.o
/usr/bin/cc     -o CMakeFiles/cmTC_ab41b.dir/CMakeCCompilerABI.c.o   -c /tmp/app/cmake-3.6.2-Linux-x86_64/share/cmake-3.6/Modules/CMakeCCompilerABI.c
Linking C executable cmTC_ab41b
/tmp/app/cmake-3.6.2-Linux-x86_64/bin/cmake -E cmake_link_script CMakeFiles/cmTC_ab41b.dir/link.txt --verbose=1
/usr/bin/cc      -v CMakeFiles/cmTC_ab41b.dir/CMakeCCompilerABI.c.o  -o cmTC_ab41b -rdynamic  
Using built-in specs.
COLLECT_GCC=/usr/bin/cc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.8.4-2ubuntu1~14.04.3' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) 
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-o' 'cmTC_ab41b' '-rdynamic' '-mtune=generic' '-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 --sysroot=/ --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -export-dynamic -dynamic-linker /lib64/ld-linux-x86-64.so.2 -z relro -o cmTC_ab41b /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. CMakeFiles/cmTC_ab41b.dir/CMakeCCompilerABI.c.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o
make[1]: Leaving directory `/tmp/app/hindsight/release/CMakeFiles/CMakeTmp'


Parsed C implicit link information from above output:
  link line regex: [^( *|.*[/\])(ld|([^/\]+-)?ld|collect2)[^/\]*( |$)]
  ignore line: [Change Dir: /tmp/app/hindsight/release/CMakeFiles/CMakeTmp]
  ignore line: []
  ignore line: [Run Build Command:"/usr/bin/make" "cmTC_ab41b/fast"]
  ignore line: [/usr/bin/make -f CMakeFiles/cmTC_ab41b.dir/build.make CMakeFiles/cmTC_ab41b.dir/build]
  ignore line: [make[1]: Entering directory `/tmp/app/hindsight/release/CMakeFiles/CMakeTmp']
  ignore line: [Building C object CMakeFiles/cmTC_ab41b.dir/CMakeCCompilerABI.c.o]
  ignore line: [/usr/bin/cc     -o CMakeFiles/cmTC_ab41b.dir/CMakeCCompilerABI.c.o   -c /tmp/app/cmake-3.6.2-Linux-x86_64/share/cmake-3.6/Modules/CMakeCCompilerABI.c]
  ignore line: [Linking C executable cmTC_ab41b]
  ignore line: [/tmp/app/cmake-3.6.2-Linux-x86_64/bin/cmake -E cmake_link_script CMakeFiles/cmTC_ab41b.dir/link.txt --verbose=1]
  ignore line: [/usr/bin/cc      -v CMakeFiles/cmTC_ab41b.dir/CMakeCCompilerABI.c.o  -o cmTC_ab41b -rdynamic  ]
  ignore line: [Using built-in specs.]
  ignore line: [COLLECT_GCC=/usr/bin/cc]
  ignore line: [COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper]
  ignore line: [Target: x86_64-linux-gnu]
  ignore line: [Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.8.4-2ubuntu1~14.04.3' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu]
  ignore line: [Thread model: posix]
  ignore line: [gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) ]
  ignore line: [COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/]
  ignore line: [LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../:/lib/:/usr/lib/]
  ignore line: [COLLECT_GCC_OPTIONS='-v' '-o' 'cmTC_ab41b' '-rdynamic' '-mtune=generic' '-march=x86-64']
  link line: [ /usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 --sysroot=/ --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -export-dynamic -dynamic-linker /lib64/ld-linux-x86-64.so.2 -z relro -o cmTC_ab41b /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. CMakeFiles/cmTC_ab41b.dir/CMakeCCompilerABI.c.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o]
    arg [/usr/lib/gcc/x86_64-linux-gnu/4.8/collect2] ==> ignore
    arg [--sysroot=/] ==> ignore
    arg [--build-id] ==> ignore
    arg [--eh-frame-hdr] ==> ignore
    arg [-m] ==> ignore
    arg [elf_x86_64] ==> ignore
    arg [--hash-style=gnu] ==> ignore
    arg [--as-needed] ==> ignore
    arg [-export-dynamic] ==> ignore
    arg [-dynamic-linker] ==> ignore
    arg [/lib64/ld-linux-x86-64.so.2] ==> ignore
    arg [-zrelro] ==> ignore
    arg [-o] ==> ignore
    arg [cmTC_ab41b] ==> ignore
    arg [/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o] ==> ignore
    arg [/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o] ==> ignore
    arg [/usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o] ==> ignore
    arg [-L/usr/lib/gcc/x86_64-linux-gnu/4.8] ==> dir [/usr/lib/gcc/x86_64-linux-gnu/4.8]
    arg [-L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu] ==> dir [/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu]
    arg [-L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib] ==> dir [/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib]
    arg [-L/lib/x86_64-linux-gnu] ==> dir [/lib/x86_64-linux-gnu]
    arg [-L/lib/../lib] ==> dir [/lib/../lib]
    arg [-L/usr/lib/x86_64-linux-gnu] ==> dir [/usr/lib/x86_64-linux-gnu]
    arg [-L/usr/lib/../lib] ==> dir [/usr/lib/../lib]
    arg [-L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../..] ==> dir [/usr/lib/gcc/x86_64-linux-gnu/4.8/../../..]
    arg [CMakeFiles/cmTC_ab41b.dir/CMakeCCompilerABI.c.o] ==> ignore
    arg [-lgcc] ==> lib [gcc]
    arg [--as-needed] ==> ignore
    arg [-lgcc_s] ==> lib [gcc_s]
    arg [--no-as-needed] ==> ignore
    arg [-lc] ==> lib [c]
    arg [-lgcc] ==> lib [gcc]
    arg [--as-needed] ==> ignore
    arg [-lgcc_s] ==> lib [gcc_s]
    arg [--no-as-needed] ==> ignore
    arg [/usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o] ==> ignore
    arg [/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o] ==> ignore
  remove lib [gcc]
  remove lib [gcc_s]
  remove lib [gcc]
  remove lib [gcc_s]
  collapse library dir [/usr/lib/gcc/x86_64-linux-gnu/4.8] ==> [/usr/lib/gcc/x86_64-linux-gnu/4.8]
  collapse library dir [/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu] ==> [/usr/lib/x86_64-linux-gnu]
  collapse library dir [/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib] ==> [/usr/lib]
  collapse library dir [/lib/x86_64-linux-gnu] ==> [/lib/x86_64-linux-gnu]
  collapse library dir [/lib/../lib] ==> [/lib]
  collapse library dir [/usr/lib/x86_64-linux-gnu] ==> [/usr/lib/x86_64-linux-gnu]
  collapse library dir [/usr/lib/../lib] ==> [/usr/lib]
  collapse library dir [/usr/lib/gcc/x86_64-linux-gnu/4.8/../../..] ==> [/usr/lib]
  implicit libs: [c]
  implicit dirs: [/usr/lib/gcc/x86_64-linux-gnu/4.8;/usr/lib/x86_64-linux-gnu;/usr/lib;/lib/x86_64-linux-gnu;/lib]
  implicit fwks: []




Detecting C [-std=c11] compiler features compiled with the following output:
Change Dir: /tmp/app/hindsight/release/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_92d6c/fast"
/usr/bin/make -f CMakeFiles/cmTC_92d6c.dir/build.make CMakeFiles/cmTC_92d6c.dir/build
make[1]: Entering directory `/tmp/app/hindsight/release/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_92d6c.dir/feature_tests.c.o
/usr/bin/cc    -std=c11 -o CMakeFiles/cmTC_92d6c.dir/feature_tests.c.o   -c /tmp/app/hindsight/release/CMakeFiles/feature_tests.c
Linking C executable cmTC_92d6c
/tmp/app/cmake-3.6.2-Linux-x86_64/bin/cmake -E cmake_link_script CMakeFiles/cmTC_92d6c.dir/link.txt --verbose=1
/usr/bin/cc       CMakeFiles/cmTC_92d6c.dir/feature_tests.c.o  -o cmTC_92d6c -rdynamic 
make[1]: Leaving directory `/tmp/app/hindsight/release/CMakeFiles/CMakeTmp'


    Feature record: C_FEATURE:1c_function_prototypes
    Feature record: C_FEATURE:1c_restrict
    Feature record: C_FEATURE:1c_static_assert
    Feature record: C_FEATURE:1c_variadic_macros


Detecting C [-std=c99] compiler features compiled with the following output:
Change Dir: /tmp/app/hindsight/release/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_ddc7f/fast"
/usr/bin/make -f CMakeFiles/cmTC_ddc7f.dir/build.make CMakeFiles/cmTC_ddc7f.dir/build
make[1]: Entering directory `/tmp/app/hindsight/release/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_ddc7f.dir/feature_tests.c.o
/usr/bin/cc    -std=c99 -o CMakeFiles/cmTC_ddc7f.dir/feature_tests.c.o   -c /tmp/app/hindsight/release/CMakeFiles/feature_tests.c
Linking C executable cmTC_ddc7f
/tmp/app/cmake-3.6.2-Linux-x86_64/bin/cmake -E cmake_link_script CMakeFiles/cmTC_ddc7f.dir/link.txt --verbose=1
/usr/bin/cc       CMakeFiles/cmTC_ddc7f.dir/feature_tests.c.o  -o cmTC_ddc7f -rdynamic 
make[1]: Leaving directory `/tmp/app/hindsight/release/CMakeFiles/CMakeTmp'


    Feature record: C_FEATURE:1c_function_prototypes
    Feature record: C_FEATURE:1c_restrict
    Feature record: C_FEATURE:0c_static_assert
    Feature record: C_FEATURE:1c_variadic_macros


Detecting C [-std=c90] compiler features compiled with the following output:
Change Dir: /tmp/app/hindsight/release/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_70c89/fast"
/usr/bin/make -f CMakeFiles/cmTC_70c89.dir/build.make CMakeFiles/cmTC_70c89.dir/build
make[1]: Entering directory `/tmp/app/hindsight/release/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_70c89.dir/feature_tests.c.o
/usr/bin/cc    -std=c90 -o CMakeFiles/cmTC_70c89.dir/feature_tests.c.o   -c /tmp/app/hindsight/release/CMakeFiles/feature_tests.c
Linking C executable cmTC_70c89
/tmp/app/cmake-3.6.2-Linux-x86_64/bin/cmake -E cmake_link_script CMakeFiles/cmTC_70c89.dir/link.txt --verbose=1
/usr/bin/cc       CMakeFiles/cmTC_70c89.dir/feature_tests.c.o  -o cmTC_70c89 -rdynamic 
make[1]: Leaving directory `/tmp/app/hindsight/release/CMakeFiles/CMakeTmp'


    Feature record: C_FEATURE:1c_function_prototypes
    Feature record: C_FEATURE:0c_restrict
    Feature record: C_FEATURE:0c_static_assert
    Feature record: C_FEATURE:0c_variadic_macros

PreinstallOutput.log:

# Run command: /tmp/app/cmake-3.6.2-Linux-x86_64/bin/cmake --build . --target "preinstall"
# Directory: /tmp/app/hindsight/release
# Output:
make: *** No rule to make target `preinstall'.  Stop.

Problems to build - CMakeFiles/hindsight_cli.dir/hs_checkpoint_reader.c.o: undefined reference to symbol 'lua_tonumber'

Hi,

I have the luasadndbox

$ dpkg -l | grep luasandbox
ii  luasandbox                                      1.2.3                                       amd64        Generic Lua sandbox for dynamic data analysis
$

But, I can't build the hindsight

$ mkdir build
$ cd build
$ cmake -DCMAKE_BUILD_TYPE=release ..
-- The C compiler identification is GNU 6.2.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/jpereira/Devel/github-mozilla/hindsight/build
$ make
Scanning dependencies of target hindsight_cli
[  3%] Building C object src/CMakeFiles/hindsight_cli.dir/hindsight.c.o
[  6%] Building C object src/CMakeFiles/hindsight_cli.dir/hs_analysis_plugins.c.o
[ 10%] Building C object src/CMakeFiles/hindsight_cli.dir/hs_checkpoint_reader.c.o
[ 13%] Building C object src/CMakeFiles/hindsight_cli.dir/hs_checkpoint_writer.c.o
[ 16%] Building C object src/CMakeFiles/hindsight_cli.dir/hs_config.c.o
[ 20%] Building C object src/CMakeFiles/hindsight_cli.dir/hs_input.c.o
[ 23%] Building C object src/CMakeFiles/hindsight_cli.dir/hs_input_plugins.c.o
[ 26%] Building C object src/CMakeFiles/hindsight_cli.dir/hs_logger.c.o
[ 30%] Building C object src/CMakeFiles/hindsight_cli.dir/hs_output.c.o
[ 33%] Building C object src/CMakeFiles/hindsight_cli.dir/hs_output_plugins.c.o
[ 36%] Building C object src/CMakeFiles/hindsight_cli.dir/hs_util.c.o
[ 40%] Linking C executable hindsight_cli
/usr/bin/ld: CMakeFiles/hindsight_cli.dir/hs_checkpoint_reader.c.o: undefined reference to symbol 'lua_tonumber'
/home/jpereira/Devel/heka-head/heka/build/heka/lib/libluasb.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
src/CMakeFiles/hindsight_cli.dir/build.make:361: recipe for target 'src/hindsight_cli' failed
make[2]: *** [src/hindsight_cli] Error 1
CMakeFiles/Makefile2:985: recipe for target 'src/CMakeFiles/hindsight_cli.dir/all' failed
make[1]: *** [src/CMakeFiles/hindsight_cli.dir/all] Error 2
Makefile:160: recipe for target 'all' failed
make: *** [all] Error 2

My setup:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.10
Release:	16.10
Codename:	yakkety
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/6/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 6.2.0-5ubuntu12' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 6.2.0 20161005 (Ubuntu 6.2.0-5ubuntu12) 
$ go version
go version go1.7.1 linux/amd64
$```

Use inotify instead of polling for dynamic plugin loading

#13

allow flexible configuration of buffering in Hindsight

Heka has a few configuration settings to control buffering:
http://hekad.readthedocs.org/en/v0.10.0b1/buffering.html#buffering-configuration-settings

It would be great to see similar configuration supported in Hindsight (for now Hindsight allows to configure output_path and output_size values: https://github.com/trink/hindsight/blob/master/docs/configuration.md#configuration-variables )

In addition to Heka's allowed values for the full_action setting (shutdown, drop, block), we could imagine a new drop_old value which would drop the oldest messages in the queue (for example dropping the oldest output file) and advance the checkpoints accordingly.

What do you think?

allow to configure random distribution of timer_events

It seems that Hindsight randomly distributes when the timer_events of the sandboxes will fire:
https://github.com/trink/hindsight/blob/master/src/hs_sandbox.c#L156-L159

I guess the goal is to better distribute the cpu load but it would be great if it could be configurable.

For example, in my current use case, I'm running tests on an analysis sandbox:
At first I dynamically load a new analysis sandbox which is configured with a ticker_interval of 3s, then inject multiple hindsight messages (through a udp json input sandbox) which should all be received by the tested analysis plugin which emits a single output message (through a udp json output sandbox) and then I check if the output message received is correct.
Because of the random distribution of timer_events, those tests sometimes fails because the multiple input messages that I send through the udp input are not all received in the same timer_event.

So it would be great if we could disable this random distribution of timer_events.

Allow an input sandbox to be stopped while blocked on a C call

e.g. A stdin input sandbox can prevent Hindsight from shutting down when blocking on a read call. I think the best option is to time out the join and cancel the thread.

can add windows support, I port it to windows, can merge it after test is pass.

can add windows support, I port it to windows, but it is test fail, can merge it after test is pass.

https://github.com/runner-mei/hindsight

Build issues if luasandbox is not installed in default system directories

I've built lua_sandbox per documentation from here.
Packed it into archive per same documentation and put everything into /tmp:

$ tree /tmp/luasandbox-1.2.1-Linux/
/tmp/luasandbox-1.2.1-Linux/
├── bin
│   └── lsb_heka_cat
├── include
│   ├── luasandbox
│   │   ├── error.h
│   │   ├── heka
│   │   │   ├── sandbox.h
│   │   │   └── stream_reader.h
│   │   ├── lauxlib.h
│   │   ├── luaconf.h
│   │   ├── lua.h
│   │   ├── lualib.h
│   │   ├── test
│   │   │   ├── mu_test.h
│   │   │   └── sandbox.h
│   │   └── util
│   │       ├── heka_message.h
│   │       ├── heka_message_matcher.h
│   │       ├── input_buffer.h
│   │       ├── output_buffer.h
│   │       ├── protobuf.h
│   │       ├── running_stats.h
│   │       ├── string.h
│   │       ├── string_matcher.h
│   │       └── util.h
│   ├── luasandbox.h
│   ├── luasandbox_output.h
│   └── luasandbox_serialize.h
├── lib64
│   ├── libluasandboxheka.so -> libluasandboxheka.so.0
│   ├── libluasandboxheka.so.0 -> libluasandboxheka.so.1.2.1
│   ├── libluasandboxheka.so.1.2.1
│   ├── libluasandbox.so -> libluasandbox.so.0
│   ├── libluasandbox.so.0 -> libluasandbox.so.1.2.1
│   ├── libluasandbox.so.1.2.1
│   ├── libluasandboxtest.so -> libluasandboxtest.so.0
│   ├── libluasandboxtest.so.0 -> libluasandboxtest.so.1.2.1
│   ├── libluasandboxtest.so.1.2.1
│   ├── libluasandboxutil.so -> libluasandboxutil.so.0
│   ├── libluasandboxutil.so.0 -> libluasandboxutil.so.1.2.1
│   └── libluasandboxutil.so.1.2.1
└── share
    ├── doc
    │   └── luasandbox
    │       └── README.md
    └── luasandbox
        └── cmake
            ├── luasandboxConfig.cmake
            └── luasandboxConfigVersion.cmake

12 directories, 37 files

Then I've tried to install hindsight per instruction from here.
In a simple case, cmake can't find luasandbox:

$ cmake -DCMAKE_BUILD_TYPE=release ..
-- The C compiler identification is GNU 6.2.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
CMake Error at CMakeLists.txt:22 (find_package):
  Could not find a package configuration file provided by "luasandbox"
  (requested version 1.2.0) with any of the following names:

    luasandboxConfig.cmake
    luasandbox-config.cmake

  Add the installation prefix of "luasandbox" to CMAKE_PREFIX_PATH or set
  "luasandbox_DIR" to a directory containing one of the above files.  If
  "luasandbox" provides a separate development package or SDK, be sure it has
  been installed.


-- Configuring incomplete, errors occurred!
See also "/home/marko/tmp/hindsight/release/CMakeFiles/CMakeOutput.log".

If I follow documentation and give cmake luasandbox dir, cmake succeeds.

$ luasandbox_DIR=/tmp/luasandbox-1.2.1-Linux cmake -DCMAKE_BUILD_TYPE=release ..
-- Configuring done
-- Generating done
-- Build files have been written to: /home/marko/tmp/hindsight/release

But make fails:

$ make
Scanning dependencies of target hindsight_cli
[  3%] Building C object src/CMakeFiles/hindsight_cli.dir/hindsight.c.o
/home/marko/tmp/hindsight/src/hindsight.c:10:32: fatal error: luasandbox/lauxlib.h: No such file or directory
 #include <luasandbox/lauxlib.h>
                                ^
compilation terminated.
src/CMakeFiles/hindsight_cli.dir/build.make:62: recipe for target 'src/CMakeFiles/hindsight_cli.dir/hindsight.c.o' failed
make[2]: *** [src/CMakeFiles/hindsight_cli.dir/hindsight.c.o] Error 1
CMakeFiles/Makefile2:985: recipe for target 'src/CMakeFiles/hindsight_cli.dir/all' failed
make[1]: *** [src/CMakeFiles/hindsight_cli.dir/all] Error 2
Makefile:160: recipe for target 'all' failed
make: *** [all] Error 2

Seems that build is bugged and it does not use given directory for luasandbox includes.

The analysis process_message is not run

I coding analysis lua like this
local cnt=0 function process_message() cnt = cnt + 1 return 0 end function timer_event() inject_payload("cbuf", "CNTS", cnt) end
But the Payload is always 0, so the process_message function cannot runs.

Race condition when using the prune_input plugin and dynamic output plugins

The pruner could delete files out from under a dynamically loaded output plugin causing read errors and stalling the plugin while it waits forever for the next input file.

Related #30

build master hindsight error

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
LIBRT_LIBRARY
linked by target "hindsight" in directory /Users/tyrionlin/opensource/hindsight/src
linked by target "hindsight_cli" in directory /Users/tyrionlin/opensource/hindsight/src
linked by target "test_config" in directory /Users/tyrionlin/opensource/hindsight/src/test

Split out checkpoint removal options

Feedback from Ops: the single setting confusing and they would at least like global configuration options at this granularity:

checkpoint_remove_on_stop = true -- default
checkpoint_remove_on_terminate = false -- default
checkpoint_remove_unused_on_startup = true -- default

see: https://github.com/trink/hindsight/pull/29

add support ssl/tls encryption to hindsight heka_tcp.lua output plugin

So that it can send data to a remote Heka instance which is configured with a TCP input with SSL/TLS.

A "Hello World" example

Are there some simple examples how to use hindsight? e.g. parsing syslog or just simple tutorial how to parse a log file and output to stdout with debugging info?

Don't allow a path component in the sandbox filename configuration

https://github.com/trink/hindsight/pull/10#commitcomment-13440510

An output_limit larger than the max_message_size should produce a configuration error

Add metrics to track the analysis thread capacity

These metrics can be used to detect a poorly balanced workload or a system that needs to be scaled out before it starts back pressuring.

hindsight does not compile anymore on ARM raspberry pi device

I get the following compilation issue:

Scanning dependencies of target hindsight
[  3%] Building C object src/CMakeFiles/hindsight.dir/hindsight.c.o
[  7%] Building C object src/CMakeFiles/hindsight.dir/hs_analysis_plugins.c.o
[ 11%] Building C object src/CMakeFiles/hindsight.dir/hs_checkpoint_reader.c.o
[ 14%] Building C object src/CMakeFiles/hindsight.dir/hs_checkpoint_writer.c.o
[ 18%] Building C object src/CMakeFiles/hindsight.dir/hs_config.c.o
[ 22%] Building C object src/CMakeFiles/hindsight.dir/hs_input.c.o
[ 25%] Building C object src/CMakeFiles/hindsight.dir/hs_input_plugins.c.o
[ 29%] Building C object src/CMakeFiles/hindsight.dir/hs_logger.c.o
[ 33%] Building C object src/CMakeFiles/hindsight.dir/hs_output.c.o
[ 37%] Building C object src/CMakeFiles/hindsight.dir/hs_output_plugins.c.o
/root/hindsight/src/hs_output_plugins.c: In function 'update_checkpoint_callback':
/root/hindsight/src/hs_output_plugins.c:70:13: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
/root/hindsight/src/hs_output_plugins.c: In function 'output_message':
/root/hindsight/src/hs_output_plugins.c:236:45: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
cc1: all warnings being treated as errors
src/CMakeFiles/hindsight.dir/build.make:261: recipe for target 'src/CMakeFiles/hindsight.dir/hs_output_plugins.c.o' failed
make[2]: *** [src/CMakeFiles/hindsight.dir/hs_output_plugins.c.o] Error 1
CMakeFiles/Makefile2:947: recipe for target 'src/CMakeFiles/hindsight.dir/all' failed
make[1]: *** [src/CMakeFiles/hindsight.dir/all] Error 2
Makefile:146: recipe for target 'all' failed
make: *** [all] Error 2

output.heka_tcp is stuck retrying a message

For some reason, my heka_tcp output plugin is stuck retrying the same message from the input queue. Here is the log message that hindsight returns: debug] output.heka_tcp retry message 32534755224530123 err: �K

And here is the complete hindsight log since the error occured for the first time:
https://framabin.org/?2b64b85db214b22b#1TG684o58DV8HenPisQSNwW/3sh1FJy1fruUqKA5wR4=

The problematic input log file is attached (6008.log) as well as the hindsight.tsv and hindsight.cp checkpoint file: hs_debug.zip

Create a CLI plugin tester

Here are some initial thoughts

heka-cat like behaviour

#18 e.g., hindsight_cli -t output -p debug.lua -i 0.log

chaining plugins to test/debug a data flow or run a report

hindsight_cli -t analysis -p a.lua -m "Type == 'foo'" -i 0.log | hindsight_cli -t output -p debug.lua -m "Type == 'a'"

cli options

-t = plugin type (input | analysis | output)
-p = plugin source
cli options for analysis and output plugins only
- -i = input file (defaults to stdin)
- -m = message matcher (defaults to "TRUE")
- -ti = ticker interval (defaults to 0)

general setup

sandbox limits will be set to unlimited
module paths will be picked up through the Lua environment variables
time will be driven by the data stream. If you run a days worth of data through in 90 seconds (and ti = 60 then the timer_event will fire 1440 times in that 90 seconds)
summary stats: performance, memory, instruction, and output limits will be written to stderr
input and analysis output goes to stdout

Document hindsight_cli

see #15

If the loader is disabled and no plugins are running HS should exit

Edge case from the discussion in #21

hindsight anaysis thread stuck because of missing input log file

I noticed that the analysis thread of my Hindsight server was stuck today, and it won't read the input log files anymore.
Finally I discovered that an input log files was missing: Hindsight was stuck because it was trying to open a new log file for reading but for some obscure reason, the log file didn't exist...
So a temporary fix was simply to touch /var/lib/hindsight/input/1745.log to create an empty file, and Hindsight analysis thread was automatically unlocked and continued its work.

I have no idea why the /var/lib/hindsight/input/1745.log input log file was missing since /var/lib/hindsight/input/1744.log and /var/lib/hindsight/input/1746.log log files were both available, and nobody has access to this server.

Any idea?

Don't panic if the 'analysis' dir is missing

If it's valid to run with an empty 'analysis' dir, it should also be valid to run with no dir at all.

hindsight output try to open "output/analysis/1.log" on every process_message

Hi,

I haven't investigated further, but it seems the output sandbox is trying to open "output/analysis/1.log" on every process_message()

a minimal reproduction config (cd in an empty dir and copy in your shell)

mkdir -p run/analysis
mkdir -p run/input
mkdir -p run/output

cat > hindsight.cfg <<'EOF'
output_path             = "output"
output_size             = 1024 * 1024 * 1024
sandbox_load_path       = nil -- disable dynamic loading
sandbox_run_path        = "run"
-- sandbox_install_path    = "/usr/share/luasandbox/sandboxes/heka" -- default

analysis_threads        = 1
analysis_lua_path       = "/usr/lib/luasandbox/modules/?.lua"
analysis_lua_cpath      = "/usr/lib/luasandbox/modules/?.so"
--io_lua_path             = analysis_lua_path ..  ";/usr/lib/luasandbox/io_modules/?.lua"
io_lua_path             = analysis_lua_path ..  ";/usr/lib/luasandbox/io_modules/?.lua"
io_lua_cpath            = analysis_lua_cpath .. ";/usr/lib/luasandbox/io_modules/?.so"

analysis_defaults = {}
input_defaults = {}
output_defaults = {}
EOF

cat > run/input/input_test.lua <<'EOF'
local msg = {
Timestamp   = nil,
Type        = "access-logs",
Hostname    = "aaaa",
Logger      = "FileTest",
Payload     = nil,
Fields      = nil
}

function process_message(offset)
    while true do
        inject_message(msg)
    end
end
EOF

cat > run/input/input_test.cfg <<'EOF' 
filename = "input_test.lua"
instruction_limit = 0
EOF

cat > run/output/counter.lua <<'EOF'
local cnt = 0

function process_message()
    cnt = cnt + 1
    return 0
end

function timer_event()
    print(cnt)
end
EOF

cat > run/output/counter.cfg <<'EOF'
filename    = "counter.lua"
message_matcher = "TRUE"
ticker_interval = 10
EOF

strace -e trace=open -f hindsight_cli hindsight.cfg 7

Thanks for hindsight!