uber / aresdb Goto Github PK

A GPU-powered real-time analytics storage and query engine.

License: Apache License 2.0

Shell 0.09% Go 85.74% Cuda 6.11% HTML 0.61% CSS 0.07% JavaScript 1.98% C 0.85% C++ 3.90% Dockerfile 0.03% CMake 0.27% ANTLR 0.37%

analytics data real-time gpu-programming database storage cuda golang query cgo

aresdb's Introduction

AresDB

AresDB is a GPU-powered real-time analytics storage and query engine. It features low query latency, high data freshness and highly efficient in-memory and on disk storage management. Please see AresDB's features, architecture design described in the Uber Engineering Blog.

This repo contains the source code of AresDB and debug UI.

Legal Note

AresDB requires the CUDA Toolkit. Please ensure you read, acknowledge, and accept the CUDA End User License Agreement.

Getting started

To get AresDB:

git clone --recursive https://github.com/uber/aresdb.git $GOPATH/src/github.com/uber/aresdb

NVIDIA Driver and CUDA Setup

AresDB needs NVIDIA driver version >= 390.48 and CUDA version 9.1.

Environment Variables

Run the following to make sure the following environment variables are correctly set:

export PATH=/path/to/cuda/bin:${PATH}
export LD_LIBRARY_PATH=/path/to/cuda/lib64:/path/to/aresdb/lib:${LD_LIBRARY_PATH}
export PKG_CONFIG_PATH=${LD_LIBRARY_PATH}/pkgconfig:${PKG_CONFIG_PATH}

Language Requirements

Building and running AresDB requires:

golang 1.11+
C++ compiler that support c++14
cmake 3.12+
nvcc version 9.1

Configure

Run following commands to generate makefile:

cmake -DQUERY_MODE=DEVICE .

Alternatively, if you want to run the query in CPU mode, run following commands:

cmake -DQUERY_MODE=HOST .

Local Test

AresDB is written in C++ (query engine) and Golang (mem store, disk store and other query components). Because of this, we break testing into two parts:

Test Golang Code

Ginkgo

We use Ginkgo as the test framework for running the Golang unit test and coverage. Install Ginkgo first and run

make test-golang

Test C++ Code

google-test

We use google-test as the test framework to test C++ code. Install google-test and set the environment variable, GTEST_ROOT, to the installed location.

After you have installed properly, run

make test-cuda

Run AresDB Server

The following command will start an AresDB server locally. You can start to query the server using a curl command or swagger page.

make run_server

Run AresDB Docker

Please read the Docker page.

Documentation

Interested in learning more about AresDB? Read the blog post

License

Apache 2.0 License, please see LICENSE for details.

aresdb's People

Contributors

Stargazers

Watchers

Forkers

natesimantov fossabot 1r053 propellingbits claudiouzelac richardji7 joe2hpimn quyphamgo sbrichardson bonedaddy biswapanda locussam shaunstanislauslau patilvikram kokes jianlirong tomzhang shafiahmed diegopacheco yyht leo23 sriharshakappala andyli029 longjohncoder micseb sjoerdapp caidongyun ariesdevil b-xiang ong tchen0123 jangocheng zmyer saromanov starters awesomegolang hadv xztl kidmam lucafuji snlpatel001213 apsaltis sizzles awesomedatatool garrusd blitzkriegcoding loreanvictor xiemaisi priestd09 ebottabi shashank734 0xdaksh blunney1 timeisapear zhanglei appcoreopc stjordanis mengjin001 codeaudit won21kr averroes wuyanxing acamtech jverce chenqin ruoshuixuelabi jiyulongxu lijingang ziyht qiangcai beaver-company chinpeng biyituan liaochy sjanulonoks jaime0815 tonylv cuulee gfaff mistshi sunnyregion lukehuang shunpeizhang pavantyagi reinhardhsu szhorizon ninollei valery-barysok nambi0915 gangk rajib2k5 radovankavicky gapdata quantiply bitpulse kumarh1982 bsmr xinding33 shaunstoltz colorfulbegin

aresdb's Issues

Improve test coverage

Describe the bug
We previously had a test coverage of 82% but now it drops to 75%. Ideally, code coverage should be > 80% for a healthy repo.

To Reproduce
Steps to reproduce the behavior:

go to https://codecov.io/gh/uber/aresdb you will see detailed overage for each package, file, method, and lines.

Expected behavior
Ideally, code coverage should be > 80% for a healthy repo.

Screenshots

upsertBatch need validate string length

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
subscriber failed to fetch enum due to timeout if a column's string length is too big.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

docker file build error

under is the last error logs

[100%] Built target npm-install
Scanning dependencies of target mem
[  5%] Building C object CMakeFiles/mem.dir/cgoutils/memory/malloc.c.o
[ 11%] Linking C shared library lib/libmem.so
[ 11%] Built target mem
Scanning dependencies of target algorithm
[ 16%] Building CUDA object CMakeFiles/algorithm.dir/query/algorithm.cu.o
[ 22%] Building CUDA object CMakeFiles/algorithm.dir/query/dimension_transform.cu.o
[ 27%] Building CUDA object CMakeFiles/algorithm.dir/query/geo_intersects.cu.o
[ 33%] Building CUDA object CMakeFiles/algorithm.dir/query/hash_reduction.cu.o
[ 44%] Building CUDA object CMakeFiles/algorithm.dir/query/functor.cu.o
[ 44%] Building CUDA object CMakeFiles/algorithm.dir/query/utils.cu.o
[ 50%] Building CUDA object CMakeFiles/algorithm.dir/query/measure_transform.cu.o
[ 55%] Building CUDA object CMakeFiles/algorithm.dir/query/hll.cu.o
[ 61%] Building CUDA object CMakeFiles/algorithm.dir/query/filter.cu.o
[ 66%] Building CUDA object CMakeFiles/algorithm.dir/query/memory.cu.o
[ 72%] Building CUDA object CMakeFiles/algorithm.dir/query/scratch_space_transform.cu.o
[ 77%] Building CUDA object CMakeFiles/algorithm.dir/query/sort_reduce.cu.o
[ 83%] Building CUDA object CMakeFiles/algorithm.dir/query/hash_lookup.cu.o
[ 88%] Building CUDA object CMakeFiles/algorithm.dir/query/transform.cu.o
/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=uint32_t]"
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(364): here

/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(128): warning: calling a __host__ function from a __host__ __device__ function is not allowed
          detected during:
            instantiation of "Facade::reference thrust::iterator_core_access::dereference(const Facade &) [with Facade=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>]"
(310): here
            instantiation of "thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::reference thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::operator*() const [with Derived=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>, Value=ares::EmptyStruct, System=thrust::host_system_tag, Traversal=thrust::random_access_traversal_tag, Reference=ares::EmptyStruct, Difference=std::ptrdiff_t]"
/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/system/detail/sequential/for_each.h(56): here
            instantiation of "InputIterator thrust::system::detail::sequential::for_each(thrust::system::detail::sequential::execution_policy<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/detail/for_each.inl(44): here
            instantiation of "InputIterator thrust::for_each(const thrust::detail::execution_policy_base<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(282): here
            instantiation of "void ares::calculateBatchIntersection(GeoShapeBatch, InputIterator, uint32_t *, int, uint32_t, uint32_t *, __nv_bool, cudaStream_t) [with InputIterator=ares::ColumnIterator<GeoPointT>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(292): here
            instantiation of "int ares::GeoIntersectionContext::run(uint32_t *, InputIterator) [with InputIterator=ares::ColumnIterator<GeoPointT>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(171): here
            instantiation of "int ares::InputVectorBinder<ares::GeoIntersectionContext, 1>::bind(InputIterators...) [with InputIterators=<>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(107): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during:
            instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int32_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(608): here
            instantiation of "ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value *, const uint32_t *, const uint32_t *, AggregateFunction) [with Value=int32_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(650): here
            instantiation of "ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value *, uint32_t *, uint32_t *, AggregateFunction) [with Value=int32_t]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(31): here
            instantiation of "int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with NInput=1, FunctorType=UnaryFunctorType]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(43): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=uint64_t]"
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(366): here

/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(128): warning: calling a __host__ function from a __host__ __device__ function is not allowed
          detected during:
            instantiation of "Facade::reference thrust::iterator_core_access::dereference(const Facade &) [with Facade=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>]"
(310): here
            instantiation of "thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::reference thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::operator*() const [with Derived=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>, Value=ares::EmptyStruct, System=thrust::host_system_tag, Traversal=thrust::random_access_traversal_tag, Reference=ares::EmptyStruct, Difference=std::ptrdiff_t]"
/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/system/detail/sequential/for_each.h(56): here
            instantiation of "InputIterator thrust::system::detail::sequential::for_each(thrust::system::detail::sequential::execution_policy<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/detail/for_each.inl(44): here
            instantiation of "InputIterator thrust::for_each(const thrust::detail::execution_policy_base<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(282): here
            instantiation of "void ares::calculateBatchIntersection(GeoShapeBatch, InputIterator, uint32_t *, int, uint32_t, uint32_t *, __nv_bool, cudaStream_t) [with InputIterator=ares::RecordIDJoinIterator<GeoPointT>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(292): here
            instantiation of "int ares::GeoIntersectionContext::run(uint32_t *, InputIterator) [with InputIterator=ares::RecordIDJoinIterator<GeoPointT>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(205): here
            instantiation of "int ares::InputVectorBinder<ares::GeoIntersectionContext, 1>::bind(InputIterators...) [with InputIterators=<>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(107): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int32_t]"
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(370): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int64_t]"
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(372): here

/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(310): warning: calling a __host__ function("ares::GeoBatchIntersectIterator< ::thrust::permutation_iterator< ::ares::VectorPartyIterator< ::GeoPointT> , unsigned int *> > ::dereference const") from a __host__ __device__ function("thrust::iterator_facade< ::ares::GeoBatchIntersectIterator< ::thrust::permutation_iterator< ::ares::VectorPartyIterator< ::GeoPointT> , unsigned int *> > ,  ::ares::EmptyStruct,  ::thrust::system::cpp::detail::tag,  ::thrust::random_access_traversal_tag,  ::ares::EmptyStruct, long> ::operator * const") is not allowed

/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(310): warning: calling a __host__ function("ares::GeoBatchIntersectIterator< ::ares::RecordIDJoinIterator< ::GeoPointT> > ::dereference const") from a __host__ __device__ function("thrust::iterator_facade< ::ares::GeoBatchIntersectIterator< ::ares::RecordIDJoinIterator< ::GeoPointT> > ,  ::ares::EmptyStruct,  ::thrust::system::cpp::detail::tag,  ::thrust::random_access_traversal_tag,  ::ares::EmptyStruct, long> ::operator * const") is not allowed

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during:
            instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=uint32_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(608): here
            instantiation of "ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value *, const uint32_t *, const uint32_t *, AggregateFunction) [with Value=uint32_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(650): here
            instantiation of "ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value *, uint32_t *, uint32_t *, AggregateFunction) [with Value=uint32_t]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(32): here
            instantiation of "int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with NInput=1, FunctorType=UnaryFunctorType]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(43): here

          detected during:
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(608): here
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(650): here
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(34): here
            instantiation of "int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with NInput=1, FunctorType=UnaryFunctorType]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(43): here

Killed
make[3]: *** [CMakeFiles/algorithm.dir/query/dimension_transform.cu.o] Error 137
CMakeFiles/algorithm.dir/build.make:75: recipe for target 'CMakeFiles/algorithm.dir/query/dimension_transform.cu.o' failed
make[3]: *** Waiting for unfinished jobs....
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = unsigned int]':
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:364:163:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = long unsigned int]':
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:366:178:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = int]':
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:370:148:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = long int]':
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:372:151:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = int]':
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:608:41:   required from 'ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value*, const uint32_t*, const uint32_t*, AggregateFunction) [with Value = int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:649:74:   required from 'ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value*, uint32_t*, uint32_t*, AggregateFunction) [with Value = int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:31:65:   required from 'int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with int NInput = 1; FunctorType = UnaryFunctorType]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:43:108:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = unsigned int]':
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:608:41:   required from 'ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value*, const uint32_t*, const uint32_t*, AggregateFunction) [with Value = unsigned int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:649:74:   required from 'ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value*, uint32_t*, uint32_t*, AggregateFunction) [with Value = unsigned int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:32:66:   required from 'int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with int NInput = 1; FunctorType = UnaryFunctorType]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:43:108:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = long int]':
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:608:41:   required from 'ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value*, const uint32_t*, const uint32_t*, AggregateFunction) [with Value = long int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:649:74:   required from 'ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value*, uint32_t*, uint32_t*, AggregateFunction) [with Value = long int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:34:65:   required from 'int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with int NInput = 1; FunctorType = UnaryFunctorType]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:43:108:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
make[2]: *** [CMakeFiles/algorithm.dir/all] Error 2
CMakeFiles/Makefile2:609: recipe for target 'CMakeFiles/algorithm.dir/all' failed
CMakeFiles/Makefile2:487: recipe for target 'CMakeFiles/aresd.dir/rule' failed
make[1]: *** [CMakeFiles/aresd.dir/rule] Error 2
make: *** [aresd] Error 2
Makefile:285: recipe for target 'aresd' failed

Useless field assignment in `query_handler.go`

Describe the bug

It looks like this assignment to aqlRequest.Verbose is dead (the field is never read in the same function, and aqlRequest is not a pointer).

To Reproduce

I found this issue using static analysis, so I cannot provide reproduction steps. I imagine one possible outcome of this could be that this if condition never holds,

Decrease recovery time

Is your feature request related to a problem? Please describe.
Takes too much time during startup to replay many redo logs

Describe the solution you'd like
Add intermittent snapshot for the fact tables

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

kaungmyat182804.wordpress

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

kaungmyat182804.wordpress

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

How to Ingest data from Apache Kafka

As mentioned in Introdution to AresDB that data can be ingested from an event bus or any streaming platform, Is there any documentation or an example on how to ingest data to the table via Apache Kafka. So far the data ingestion part is not so clear from the available resources.

refactor error handling

Is your feature request related to a problem? Please describe.
user vs. system error

Error in build docker 96% finished [Built target arrow_cuda_objlib]

under is the last part of building process

[ 94%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/util/trie.cc.o
[ 95%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/util/logging.cc.o
[ 96%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/ipc/json-internal.cc.o
[ 96%] Built target arrow_cuda_objlib
Scanning dependencies of target arrow_cuda_static
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:1001: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/ipc/json-simple.cc.o' failed
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/ipc/json-simple.cc.o] Error 4
make[5]: *** Waiting for unfinished jobs....
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:780: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/util/cpu-info.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/util/cpu-info.cc.o] Error 4
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/io/buffered.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:624: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/io/buffered.cc.o' failed
[ 97%] Linking CXX static library ../../../release/libarrow_cuda.a
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:858: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/util/memory.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/util/memory.cc.o] Error 4
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:845: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/util/key_value_metadata.cc.o' failed
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/util/key_value_metadata.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:923: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/util/utf8.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/util/utf8.cc.o] Error 4
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/filesystem/mockfs.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:507: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/filesystem/mockfs.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:741: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/util/basic_decimal.cc.o' failed
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/util/basic_decimal.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:663: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/io/hdfs.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/io/hdfs.cc.o] Error 4
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/type.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:377: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/type.cc.o' failed
/usr/bin/ar qc ../../../release/libarrow_cuda.a CMakeFiles/arrow_cuda_objlib.dir/cuda_arrow_ipc.cc.o CMakeFiles/arrow_cuda_objlib.dir/cuda_context.cc.o CMakeFiles/arrow_cuda_objlib.dir/cuda_memory.cc.o
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/record_batch.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:273: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/record_batch.cc.o' failed
/usr/bin/ranlib ../../../release/libarrow_cuda.a
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:637: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/io/compressed.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/io/compressed.cc.o] Error 4
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:754: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/util/bit-util.cc.o' failed
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/util/bit-util.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:143: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_dict.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_dict.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:104: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_base.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_base.cc.o] Error 4
[ 97%] Built target arrow_cuda_static
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:728: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/testing/util.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/testing/util.cc.o] Error 4
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_primitive.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:169: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_primitive.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/util/int-util.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:806: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/util/int-util.cc.o' failed
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:650: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/io/file.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/io/file.cc.o] Error 4
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_adaptive.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:91: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_adaptive.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_nested.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:156: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_nested.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/util/decimal.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:793: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/util/decimal.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/filesystem/util-internal.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:533: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/filesystem/util-internal.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/table.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:338: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/table.cc.o' failed
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:1027: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/ipc/metadata-internal.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/ipc/metadata-internal.cc.o] Error 4
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/buffer.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:208: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/buffer.cc.o' failed
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:195: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/array/concatenate.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/array/concatenate.cc.o] Error 4
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/csv/converter.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:403: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/csv/converter.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/io/interfaces.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:689: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/io/interfaces.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:936: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/vendored/datetime/tz.cpp.o' failed
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/vendored/datetime/tz.cpp.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:585: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/json/converter.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/json/converter.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:221: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/compare.cc.o' failed
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:468: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/csv/reader.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/compare.cc.o] Error 4
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/csv/reader.cc.o] Error 4
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/util/trie.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:910: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/util/trie.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_decimal.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:130: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_decimal.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/table_builder.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:351: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/table_builder.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/filesystem/path-util.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:520: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/filesystem/path-util.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/json/chunker.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:572: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/json/chunker.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/io/readahead.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:715: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/io/readahead.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/csv/column-builder.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:429: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/csv/column-builder.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:182: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_union.cc.o' failed
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/array/builder_union.cc.o] Error 4
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/json/parser.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:598: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/json/parser.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/pretty_print.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:260: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/pretty_print.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/sparse_tensor.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:312: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/sparse_tensor.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/io/hdfs-internal.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:676: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/io/hdfs-internal.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/ipc/message.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:1014: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/ipc/message.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:1053: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/ipc/writer.cc.o' failed
g++-5: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/json/reader.cc.o] Error 4
src/arrow/CMakeFiles/arrow_objlib.dir/build.make:611: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/json/reader.cc.o' failed
make[4]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/all] Error 2
make[3]: *** [all] Error 2
CMakeFiles/Makefile2:554: recipe for target 'src/arrow/CMakeFiles/arrow_objlib.dir/all' failed
Makefile:143: recipe for target 'all' failed
CMakeFiles/Arrow.dir/build.make:111: recipe for target 'Arrow-prefix/src/Arrow-stamp/Arrow-build' failed
make[2]: *** [Arrow-prefix/src/Arrow-stamp/Arrow-build] Error 2
CMakeFiles/Makefile2:75: recipe for target 'CMakeFiles/Arrow.dir/all' failed
make[1]: *** [CMakeFiles/Arrow.dir/all] Error 2
make: *** [all] Error 2
Makefile:83: recipe for target 'all' failed
CMake Error at cmake/Modules/ConfigureArrow.cmake:73 (message):
Building Arrow failed: 2
Call Stack (most recent call first):
CMakeLists.txt:118 (include)
-- Configuring incomplete, errors occurred!
See also "/cudf/cpp/build/CMakeFiles/CMakeOutput.log".
Removing intermediate container 49fbffde0810
The command '/bin/bash -c source activate cudf && mkdir -p /cudf/cpp/build && cd /cudf/cpp/build && cmake .. -DCMAKE_INSTALL_PREFIX=${CONDA_PREFIX} && make -j install' returned a non-zero code: 1

Simplify c++ binder to pass in data type as parameter which using pointer to pass data

Is your feature request related to a problem? Please describe.
Compilation takes a long time due to the many data type binding in the template, also it is hard to add new data type, need to change multiple places with specialized template

Describe the solution you'd like
Pass data type and value pointer in context/functor/, data conversion in realtime according to the data type.

schema creation with default values and validation of configs

Describe the bug
currently, user create schema through api and did not fill in table configs will produce invalid table schema and cause server panic once trying to ingest data until server restart

To Reproduce

create table through api without table config
ingest data
restart server

Expected behavior
panic at step 2
return normal at step 3

Avg aggregate support

Is your feature request related to a problem? Please describe.
currently AresDB does not directly support average aggregate, user will have to compute average based on sum and count aggregate

Describe the solution you'd like
support average aggregate in query engine

Describe alternatives you've considered

Additional context

create hll column with hll config for fast hll calculation

Is your feature request related to a problem? Please describe.
Currently we support fast hyperloglog calculation based on pre computed hll column which requires users to create and ingest.

Describe the solution you'd like

Schema: when user create a column and specify hyperloglog in the column config, we should automatically create the hyperloglog column
Ingestion: ingestion into the hyperloglog enabled column should automatically create hll value and ingest into the hll column

Geo intersection behaves incorrectly when number of points * number of rows is too big.

Describe the bug
Geo intersection behaves incorrectly when number of points * number of rows is too big.
When adding more geos, the result size decreases.

To Reproduce
query:
use the following query

{
  "queries": [
    {
      "dimensions": [
        {
          "sqlExpression": "trips.request_at",
          "timeBucketizer": "day",
          "timeUnit" :"second"
        }
      ],
      "joins": [
        {
          "alias": "g",
          "table": "geofences_configstore_udr_geofences",
          "conditions": [
            "geography_intersects(g.shape, request_point)"
          ]
        }
      ],
      "measures": [
        {
          "sqlExpression": "count(*)"
        }
      ],
      "rowFilters": [
        "status = 'completed'",
        "g.geofence_uuid IN (0x2370A118017844258B5B0CA2AA24058A, 0x451F86CC12A543F4B3F795F247B6EF21, 0xAAE54A49FC1D43E0BB138F0AA2D0F28B, 0x7F9935CD9620475BA388E37194CE9AF6, 0xC2936F2AACAB4A2C936DCE75E23876A4)"
      ],
      "table": "trips",
      "timeFilter": {
        "column": "request_at",
        "from": "-7d",
        "to": "now"
      }
    }
  ]
}

Expected behavior
Count should increase after adding more geos.

Error populating query using the examples

Describe the bug
I was following the examples given in aresdb/examples to populate the database with the tables, data and queries. Both the table and data worked fine, but the example for the query resulted in panic: query failed with status code 400. I have attached the full error here: error.log.

To Reproduce
Follow the example code given at aresdb/examples:

cd examples
go build examples.go
./examples tables --dataset 1k_trips
./examples data --dataset 1k_trips
./examples query --dataset 1k_trips

Expected behavior
Expected the database to respond with the response from the queries.

Desktop (please complete the following information):

OS: Ubuntu 18.04.4 LTS
Browser: Firefox 76.0.1
Docker: v19.03.11
nvidia-docker: v19.03.11

Additional context
I am using the docker image of the AresDB.

replace rdkafka with sarama to mitigate compiling issue

Is your feature request related to a problem? Please describe.
The compiler spent a lot of time on the compilation of rdkafka and sometimes stuck

Describe the solution you'd like
Replace rdkafka with simpler kafka consumer like Sarama

`defer` in the query_handler for loop

I've just start looking how query handler is build and there is a defer in the for loop.
It does not look like a feature.

Steps to reproduce the behavior:

Go to https://github.com/uber/aresdb/blob/master/api/query_handler.go#L177

AresDB query client library

Is your feature request related to a problem? Please describe.
currently user manually format query in json format and send over http

Describe the solution you'd like
Create client library to hide query composing logic with version control

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Build an interactive shell application

Is your feature request related to a problem? Please describe.
Right now AresDB provides a swagger page to send requests to AresDB but it's not intuitive to use.

Describe the solution you'd like
A better way may be is to provide an interactive shell application to talk to AresDB including make the query, schema changes, and sample data ingestion.
We may consider calling this binary "ares" and original aresdb binary to be "aresd"

Failed to use swagger in docker

Describe the bug
Failed to use swagger in docker

To Reproduce

Follow: https://github.com/uber/aresdb/blob/master/docker/README.md
Visit http://my-ip:9374/swagger/ (my-ip is the docker host's ip)

Chrome console output:

Refused to apply style from 'http://my-ip:9374/node_modules/swagger-ui-dist/swagger-ui.css' because its MIME type ('text/plain') is not a supported stylesheet MIME type, and strict MIME checking is enabled.
(index):14 GET http://my-ip:9374/node_modules/swagger-ui-dist/swagger-ui-bundle.js net::ERR_ABORTED 404 (Not Found)
(index):15 GET http://my-ip:9374/node_modules/swagger-ui-dist/swagger-ui-standalone-preset.js net::ERR_ABORTED 404 (Not Found)

fix debug ui dependencies in docker

Describe the bug
debug ui failed to load DataTable in docker

To Reproduce
Steps to reproduce the behavior:
launch docker
navigate to debug ui

CUDA DeviceFree: out of memory error when building Aresdb first time

Describe the issue
When running 'make run_server' to build version 0.0.2, the build fails with a DeviceFree: out of memory error after a few minutes. I am using a new server with no other processes running.

Reproduce the issue
NVIDIA driver version: 390.48
Cuda version: release 9.1, V9.1.85
golang version: 1.13
gcc version: 5.4.0
cmake version: 3.15.4

Follow the instructions to compile Aresdb version 0.0.2 through 'run make_server'

Error message
[ 15%] Built target mem
[100%] Built target algorithm
[100%] Built target lib
[100%] Built target aresd
Using config file: config/ares.yaml
{"level":"info","msg":"Bootstrapping service","config":{"Port":9374,"DebugPort":43202,"RootPath":"ares-root","TotalMemorySize":161061273600,"SchedulerOff":false,"Version":"","Env":"","Query":{"DeviceMemoryUtilization":0.95,"DeviceChoosingTimeout":10,"TimezoneTable":{"TableName":"api_cities"},"EnableHashReduction":false},"DiskStore":{"WriteSync":true},"HTTP":{"MaxConnections":300,"ReadTimeOutInSeconds":20,"WriteTimeOutInSeconds":300},"RedoLogConfig":{"DiskConfig":{"Disabled":false},"KafkaConfig":{"Enabled":false,"Brokers":null,"TopicSuffix":""},"DiskOnlyForUnsharded":false},"Cluster":{"Enable":false,"Distributed":false,"Namespace":"","InstanceID":"","Controller":{"Address":"localhost:6708","Headers":null,"TimeoutSec":0},"Etcd":{"Zone":"local","Env":"dev","Service":"ares-datanode","CacheDir":"","ETCDClusters":[{"Zone":"local","Endpoints":["127.0.0.1:2379"],"KeepAlive":null,"TLS":null}],"SDConfig":{"InitTimeout":null},"WatchWithRevision":0},"HeartbeatConfig":{"Timeout":10,"Interval":1}}}}
panic: ERROR when calling CUDA functions: DeviceFree: out of memory

goroutine 1 [running]:
github.com/uber/aresdb/utils.StackError(0x0, 0x0, 0xc00004e040, 0x3d, 0x0, 0x0, 0x0, 0x0)
        /nvme1n1/go1/src/github.com/uber/aresdb/utils/error.go:61 +0x3f9
github.com/uber/aresdb/cgoutils.DoCGoCall(0xc0005b2e18, 0xc0004a44d0)
        /nvme1n1/go1/src/github.com/uber/aresdb/cgoutils/utils.go:31 +0xa7
github.com/uber/aresdb/cgoutils.doCGoCall(0xc0005b2e48, 0x1)
        /nvme1n1/go1/src/github.com/uber/aresdb/cgoutils/memory.go:188 +0x49
github.com/uber/aresdb/cgoutils.DeviceFree(0x0, 0x0)
        /nvme1n1/go1/src/github.com/uber/aresdb/cgoutils/memory.go:111 +0x5c
github.com/uber/aresdb/cmd/aresd/cmd.start(0x249e, 0xa8c2, 0xc0005660c0, 0x9, 0x2580000000, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
       /nvme1n1/go1/src/github.com/uber/aresdb/cmd/aresd/cmd/cmd.go:103 +0x1c2
github.com/uber/aresdb/cmd/aresd/cmd.Execute.func1(0xc00038e000, 0x1e39648, 0x0, 0x0)
        /nvme1n1/go1/src/github.com/uber/aresdb/cmd/aresd/cmd/cmd.go:85 +0x13d
github.com/spf13/cobra.(*Command).execute(0xc00038e000, 0xc00003c1d0, 0x0, 0x0, 0xc00038e000, 0xc00003c1d0)
        /nvme1n1/go1/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xc00038e000, 0xc0004a2050, 0x5, 0x134fe40)
        /nvme1n1/go1/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
        /nvme1n1/go1/pkg/mod/github.com/spf13/[email protected]/command.go:864
github.com/uber/aresdb/cmd/aresd/cmd.Execute(0x0, 0x0, 0x0)
        /nvme1n1/go1/src/github.com/uber/aresdb/cmd/aresd/cmd/cmd.go:95 +0x229
main.main()
        /nvme1n1/go1/src/github.com/uber/aresdb/cmd/aresd/main.go:20 +0x32

goroutine 1 [running]:
github.com/uber/aresdb/cgoutils.DoCGoCall(0xc0005b2e18, 0xc0004a44d0)
        /nvme1n1/go1/src/github.com/uber/aresdb/cgoutils/utils.go:31 +0xc1
github.com/uber/aresdb/cgoutils.doCGoCall(0xc0005b2e48, 0x1)
        /nvme1n1/go1/src/github.com/uber/aresdb/cgoutils/memory.go:188 +0x49
github.com/uber/aresdb/cgoutils.DeviceFree(0x0, 0x0)
        /nvme1n1/go1/src/github.com/uber/aresdb/cgoutils/memory.go:111 +0x5c
github.com/uber/aresdb/cmd/aresd/cmd.start(0x249e, 0xa8c2, 0xc0005660c0, 0x9, 0x2580000000, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /nvme1n1/go1/src/github.com/uber/aresdb/cmd/aresd/cmd/cmd.go:103 +0x1c2
github.com/uber/aresdb/cmd/aresd/cmd.Execute.func1(0xc00038e000, 0x1e39648, 0x0, 0x0)
        /nvme1n1/go1/src/github.com/uber/aresdb/cmd/aresd/cmd/cmd.go:85 +0x13d
github.com/spf13/cobra.(*Command).execute(0xc00038e000, 0xc00003c1d0, 0x0, 0x0, 0xc00038e000, 0xc00003c1d0)
        /nvme1n1/go1/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xc00038e000, 0xc0004a2050, 0x5, 0x134fe40)
        /nvme1n1/go1/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
        /nvme1n1/go1/pkg/mod/github.com/spf13/[email protected]/command.go:864
github.com/uber/aresdb/cmd/aresd/cmd.Execute(0x0, 0x0, 0x0)
        /nvme1n1/go1/src/github.com/uber/aresdb/cmd/aresd/cmd/cmd.go:95 +0x229
main.main()
        /nvme1n1/go1/src/github.com/uber/aresdb/cmd/aresd/main.go:20 +0x32
CMakeFiles/run_server.dir/build.make:57: recipe for target 'CMakeFiles/run_server' failed
make[3]: *** [CMakeFiles/run_server] Error 2
CMakeFiles/Makefile2:467: recipe for target 'CMakeFiles/run_server.dir/all' failed
make[2]: *** [CMakeFiles/run_server.dir/all] Error 2
CMakeFiles/Makefile2:474: recipe for target 'CMakeFiles/run_server.dir/rule' failed
make[1]: *** [CMakeFiles/run_server.dir/rule] Error 2
Makefile:298: recipe for target 'run_server' failed
make: *** [run_server] Error 2

docker build error

docker build error in last step
RUN make aresd -j

Step 27/27 : RUN make aresd -j
 ---> Running in 2bdfb1ea1d1d
Scanning dependencies of target mem
[  5%] Building C object CMakeFiles/mem.dir/cgoutils/memory/malloc.c.o
[ 11%] Linking C shared library lib/libmem.so
[ 11%] Built target mem
Scanning dependencies of target algorithm
[ 16%] Building CUDA object CMakeFiles/algorithm.dir/query/algorithm.cu.o
[ 22%] Building CUDA object CMakeFiles/algorithm.dir/query/dimension_transform.cu.o
[ 27%] Building CUDA object CMakeFiles/algorithm.dir/query/filter.cu.o
[ 33%] Building CUDA object CMakeFiles/algorithm.dir/query/functor.cu.o
[ 38%] Building CUDA object CMakeFiles/algorithm.dir/query/geo_intersects.cu.o
[ 44%] Building CUDA object CMakeFiles/algorithm.dir/query/hash_lookup.cu.o
[ 50%] Building CUDA object CMakeFiles/algorithm.dir/query/hash_reduction.cu.o
[ 55%] Building CUDA object CMakeFiles/algorithm.dir/query/hll.cu.o
[ 61%] Building CUDA object CMakeFiles/algorithm.dir/query/memory.cu.o
[ 66%] Building CUDA object CMakeFiles/algorithm.dir/query/measure_transform.cu.o
[ 72%] Building CUDA object CMakeFiles/algorithm.dir/query/scratch_space_transform.cu.o
[ 77%] Building CUDA object CMakeFiles/algorithm.dir/query/sort_reduce.cu.o
[ 83%] Building CUDA object CMakeFiles/algorithm.dir/query/transform.cu.o
[ 88%] Building CUDA object CMakeFiles/algorithm.dir/query/utils.cu.o
/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during:
            instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int32_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(608): here
            instantiation of "ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value *, const uint32_t *, const uint32_t *, AggregateFunction) [with Value=int32_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(650): here
            instantiation of "ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value *, uint32_t *, uint32_t *, AggregateFunction) [with Value=int32_t]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(31): here
            instantiation of "int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with NInput=1, FunctorType=UnaryFunctorType]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(43): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=uint32_t]"
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(364): here

/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(128): warning: calling a __host__ function from a __host__ __device__ function is not allowed
          detected during:
            instantiation of "Facade::reference thrust::iterator_core_access::dereference(const Facade &) [with Facade=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>]"
(310): here
            instantiation of "thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::reference thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::operator*() const [with Derived=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>, Value=ares::EmptyStruct, System=thrust::host_system_tag, Traversal=thrust::random_access_traversal_tag, Reference=ares::EmptyStruct, Difference=std::ptrdiff_t]"
/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/system/detail/sequential/for_each.h(56): here
            instantiation of "InputIterator thrust::system::detail::sequential::for_each(thrust::system::detail::sequential::execution_policy<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/detail/for_each.inl(44): here
            instantiation of "InputIterator thrust::for_each(const thrust::detail::execution_policy_base<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(282): here
            instantiation of "void ares::calculateBatchIntersection(GeoShapeBatch, InputIterator, uint32_t *, int, uint32_t, uint32_t *, __nv_bool, cudaStream_t) [with InputIterator=ares::ColumnIterator<GeoPointT>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(292): here
            instantiation of "int ares::GeoIntersectionContext::run(uint32_t *, InputIterator) [with InputIterator=ares::ColumnIterator<GeoPointT>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(171): here
            instantiation of "int ares::InputVectorBinder<ares::GeoIntersectionContext, 1>::bind(InputIterators...) [with InputIterators=<>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(107): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=uint64_t]"
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(366): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int32_t]"
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(370): here

/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(128): warning: calling a __host__ function from a __host__ __device__ function is not allowed
          detected during:
            instantiation of "Facade::reference thrust::iterator_core_access::dereference(const Facade &) [with Facade=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>]"
(310): here
            instantiation of "thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::reference thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::operator*() const [with Derived=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>, Value=ares::EmptyStruct, System=thrust::host_system_tag, Traversal=thrust::random_access_traversal_tag, Reference=ares::EmptyStruct, Difference=std::ptrdiff_t]"
/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/system/detail/sequential/for_each.h(56): here
            instantiation of "InputIterator thrust::system::detail::sequential::for_each(thrust::system::detail::sequential::execution_policy<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/detail/for_each.inl(44): here
            instantiation of "InputIterator thrust::for_each(const thrust::detail::execution_policy_base<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(282): here
            instantiation of "void ares::calculateBatchIntersection(GeoShapeBatch, InputIterator, uint32_t *, int, uint32_t, uint32_t *, __nv_bool, cudaStream_t) [with InputIterator=ares::RecordIDJoinIterator<GeoPointT>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(292): here
            instantiation of "int ares::GeoIntersectionContext::run(uint32_t *, InputIterator) [with InputIterator=ares::RecordIDJoinIterator<GeoPointT>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(205): here
            instantiation of "int ares::InputVectorBinder<ares::GeoIntersectionContext, 1>::bind(InputIterators...) [with InputIterators=<>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(107): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during:
            instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=uint32_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(608): here
            instantiation of "ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value *, const uint32_t *, const uint32_t *, AggregateFunction) [with Value=uint32_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(650): here
            instantiation of "ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value *, uint32_t *, uint32_t *, AggregateFunction) [with Value=uint32_t]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(32): here
            instantiation of "int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with NInput=1, FunctorType=UnaryFunctorType]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(43): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int64_t]"
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(372): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during:
            instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int64_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(608): here
            instantiation of "ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value *, const uint32_t *, const uint32_t *, AggregateFunction) [with Value=int64_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(650): here
            instantiation of "ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value *, uint32_t *, uint32_t *, AggregateFunction) [with Value=int64_t]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(34): here
            instantiation of "int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with NInput=1, FunctorType=UnaryFunctorType]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(43): here

/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(310): warning: calling a __host__ function("ares::GeoBatchIntersectIterator< ::thrust::permutation_iterator< ::ares::VectorPartyIterator< ::GeoPointT> , unsigned int *> > ::dereference const") from a __host__ __device__ function("thrust::iterator_facade< ::ares::GeoBatchIntersectIterator< ::thrust::permutation_iterator< ::ares::VectorPartyIterator< ::GeoPointT> , unsigned int *> > ,  ::ares::EmptyStruct,  ::thrust::system::cpp::detail::tag,  ::thrust::random_access_traversal_tag,  ::ares::EmptyStruct, long> ::operator * const") is not allowed

/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(310): warning: calling a __host__ function("ares::GeoBatchIntersectIterator< ::ares::RecordIDJoinIterator< ::GeoPointT> > ::dereference const") from a __host__ __device__ function("thrust::iterator_facade< ::ares::GeoBatchIntersectIterator< ::ares::RecordIDJoinIterator< ::GeoPointT> > ,  ::ares::EmptyStruct,  ::thrust::system::cpp::detail::tag,  ::thrust::random_access_traversal_tag,  ::ares::EmptyStruct, long> ::operator * const") is not allowed

/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = unsigned int]':
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:364:163:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = long unsigned int]':
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:366:178:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = int]':
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:370:148:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = long int]':
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:372:151:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = int]':
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:608:41:   required from 'ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value*, const uint32_t*, const uint32_t*, AggregateFunction) [with Value = int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:649:74:   required from 'ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value*, uint32_t*, uint32_t*, AggregateFunction) [with Value = int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:31:65:   required from 'int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with int NInput = 1; FunctorType = UnaryFunctorType]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:43:108:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = unsigned int]':
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:608:41:   required from 'ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value*, const uint32_t*, const uint32_t*, AggregateFunction) [with Value = unsigned int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:649:74:   required from 'ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value*, uint32_t*, uint32_t*, AggregateFunction) [with Value = unsigned int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:32:66:   required from 'int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with int NInput = 1; FunctorType = UnaryFunctorType]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:43:108:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = long int]':
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:608:41:   required from 'ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value*, const uint32_t*, const uint32_t*, AggregateFunction) [with Value = long int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:649:74:   required from 'ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value*, uint32_t*, uint32_t*, AggregateFunction) [with Value = long int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:34:65:   required from 'int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with int NInput = 1; FunctorType = UnaryFunctorType]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:43:108:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]


[ 94%] Linking CUDA device code CMakeFiles/algorithm.dir/cmake_device_link.o
[100%] Linking CUDA shared library lib/libalgorithm.so
[100%] Built target algorithm
Scanning dependencies of target lib
[100%] Built target lib
Scanning dependencies of target aresd
make[3]: go: Command not found
CMakeFiles/aresd.dir/build.make:480: recipe for target 'CMakeFiles/aresd' failed
make[3]: *** [CMakeFiles/aresd] Error 127
CMakeFiles/Makefile2:546: recipe for target 'CMakeFiles/aresd.dir/all' failed
make[2]: *** [CMakeFiles/aresd.dir/all] Error 2
CMakeFiles/Makefile2:553: recipe for target 'CMakeFiles/aresd.dir/rule' failed
make[1]: *** [CMakeFiles/aresd.dir/rule] Error 2
Makefile:311: recipe for target 'aresd' failed
make: *** [aresd] Error 2
The command '/bin/sh -c make aresd -j' returned a non-zero code: 2

Add documentation, usage examples, and client/drivers

Do you intend to provide documentation, usage examples, and clients/drivers for various programming/REST/GraphQL languages, etc.?

Skip compilation of cuda lib if no c/cpp code changes

It will help us reduce the build time

Coverage integration

https://docs.travis-ci.com/user/coveralls/

How to implement a common selection query

In the 'Ares Query Language' document, the last example describes that "While AQL is designed primarily for time series aggregates, it is also possible to express non-aggregate queries, such as selection queries." but when i Use 1 or any constant as the measure expression, it can not run. I want to know how to implement a common selection query.can you give me a example or a detailed document about the Ares Query Language.

Useless field assignment in `aql_nonaggr_batchexecutor.go`

Describe the bug

It looks like this assignment to bc.size is dead (bc is a local variable which does not escape, and the field is not read anywhere).

To Reproduce

I found this issue using static analysis, so I cannot provide reproduction steps. I think this probably has no observable effect in practice.

Logger and metrics in c++ code

Is your feature request related to a problem? Please describe.
Right now inside c++ code we don't have any logging or metric reporting. However, this is very useful. e.g. report gridDim, blockDim, etc.

Describe the solution you'd like
One thing we can try is to provide a golang call back to c call

Describe alternatives you've considered
Or reimplement the logic in c++

Base docker and examples throws error at data ingest "panic: Missing primary key column"

Describe the bug
A clear and concise description of what the bug is.
Running basic docker and examples throws error:
panic: Missing primary key column

To Reproduce
Steps to reproduce the behavior:
Follow docker instructions
Follow example instructions inside docker container

Expected behavior
A clear and concise description of what you expected to happen.
Be able to run examples in docker container, so I can get to a place where I can query data via swagger interface. Would also be useful to have a plain description of how to get data ingested via swagger interface as trying to dig through go code inside a docker container to trouble shoot is cumbersome.

Full details=============
s@a:$ sudo nvidia-docker run -p 9374:9374/tcp -p 43202:43202/tcp -it aresdb:latest
[sudo] password for s:
root@020b4c0afe0a:/go/src/github.com/uber/aresdb# make run_server >> ./log/aresdb.log 2>&1 &
[1] 15
root@020b4c0afe0a:/go/src/github.com/uber/aresdb# cd examples/
root@020b4c0afe0a:/go/src/github.com/uber/aresdb/examples# go build examples.go
root@020b4c0afe0a:/go/src/github.com/uber/aresdb/examples# ./examples tables --dataset 1k_trips
table cities created
table trips created
root@020b4c0afe0a:/go/src/github.com/uber/aresdb/examples# ./examples data --dataset 1k_trips
panic: Missing primary key column
goroutine 1 [running]:
github.com/uber/aresdb/utils.StackError(0x0, 0x0, 0x8f01c9, 0x1a, 0x0, 0x0, 0x0, 0x0)
/root/go/src/github.com/uber/aresdb/utils/error.go:61 +0x46e
github.com/uber/aresdb/client.checkPrimaryKeys(0xc42033a140, 0xc420316ce0, 0x2, 0x2, 0x0, 0x0)
/root/go/src/github.com/uber/aresdb/client/connector.go:241 +0x14b
github.com/uber/aresdb/client.(*connector).prepareUpsertBatch(0xc42026c9a0, 0xc4200149f0, 0x6, 0xc420316ce0, 0x2, 0x2, 0xc420018fb0, 0x2, 0x2, 0xc4203a6000, ...)
/root/go/src/github.com/uber/aresdb/client/connector.go:223 +0xae
github.com/uber/aresdb/client.(*connector).Insert(0xc42026c9a0, 0xc4200149f0, 0x6, 0xc420316ce0, 0x2, 0x2, 0xc4203a6000, 0xa, 0x10, 0x0, ...)
/root/go/src/github.com/uber/aresdb/client/connector.go:169 +0x247
main.ingestDataForTable(0xb3b020, 0xc42026c9a0, 0xc4200149f0, 0x6, 0xc420014d00, 0x1a)
/root/go/src/github.com/uber/aresdb/examples/examples.go:187 +0x961
main.ingestDataForDataSet()
/root/go/src/github.com/uber/aresdb/examples/examples.go:115 +0x370
main.main.func1(0xc4200af900, 0xc420316740, 0x0, 0x2)
/root/go/src/github.com/uber/aresdb/examples/examples.go:214 +0x20
github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).execute(0xc4200af900, 0xc4203166e0, 0x2, 0x2, 0xc4200af900, 0xc4203166e0)
/root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:766 +0x2c1
github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200af680, 0x8e1260, 0xc4200af7d0, 0xc42014df68)
/root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:852 +0x334
github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200af680, 0xc42014df58, 0x3)
/root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
/root/go/src/github.com/uber/aresdb/examples/examples.go:239 +0x524

goroutine 1 [running]:
main.panicIfErr(0xb3b0a0, 0xc4203193e0)
/root/go/src/github.com/uber/aresdb/examples/examples.go:67 +0x4a
main.ingestDataForTable(0xb3b020, 0xc42026c9a0, 0xc4200149f0, 0x6, 0xc420014d00, 0x1a)
/root/go/src/github.com/uber/aresdb/examples/examples.go:188 +0x986
main.ingestDataForDataSet()
/root/go/src/github.com/uber/aresdb/examples/examples.go:115 +0x370
main.main.func1(0xc4200af900, 0xc420316740, 0x0, 0x2)
/root/go/src/github.com/uber/aresdb/examples/examples.go:214 +0x20
github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).execute(0xc4200af900, 0xc4203166e0, 0x2, 0x2, 0xc4200af900, 0xc4203166e0)
/root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:766 +0x2c1
github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200af680, 0x8e1260, 0xc4200af7d0, 0xc42014df68)
/root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:852 +0x334
github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200af680, 0xc42014df58, 0x3)
/root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
/root/go/src/github.com/uber/aresdb/examples/examples.go:239 +0x524
root@020b4c0afe0a:~/go/src/github.com/uber/aresdb/examples#

AresDB on Arm/Arm64

Has anyone got AresDB built for Arm? I think AresDB would be powerful on the Jetson platform.

Use cache to store dependencies

candidates are:

cuda
cppcheck and cpplint

API versioning

Is your feature request related to a problem? Please describe.
currently our api is not versioned, this leads to risk in API updates, eg. ingestion upsert batch update or query response update

Describe the solution you'd like
Common api versioning strategies:

Version by URL path (ex: /api/v1/foo)
Version by URL query string (ex: /api/foo?api-version=1.0)
Version by HTTP header (ex: api-version: 1.0)
Version by media type negotiation (ex: accept: application/vnd.mycompany.myapp-v2+xml)

lint on only changed files

Right now make lint is working on all source code, we can provide another command to lint on only changed files, maybe only for running locally.

Create ares docker image

To help users quickly set up an environment with all "batteries included", we can provide a docker image with all dependencies pre-installed so that user can simply run a few commands to query ares

Support dplyr or nodbi R client

aresdb looks very promising, welcome to wrap aresdb in the R ecosystem.

wish the aresdb can be a PostGIS or a mysql alternative for raster query in R.

Currently, NoSQL can wrapped by nodbi and SQL can wrapped
by dplyr

Update services.Service with placement.Placement in m3cluster

Hi,

I noticed you guys are using the services.Service interface from m3cluster, I would like to recommend you guys to switch to placement.Placement instead unless you need to filter by healthy/unhealthy nodes in the placement.

The Service interface is a thin wrapper on top of Placement interface and the main goal is to filter out unhealthy nodes but that behavior ended up not being used in m3, as you can see more operations has been built on Placement and PlacementService interface than service.Service (like MarkShardsAvailable etc).

I hope one day we can migrate m3 to placement.Placement only and deprecate services.Service, and removing external dependencies on services.Service would help reaching that.

Let me know if you have any concerns.

Thanks,
Chao

Ares Connector: Multiple table points to the same schema

Describe the bug
Ares Connector caching the schema mapping, however multiple table points to the same schema object.

for _, table := range tables {
		cf.setTable(&table)
		err := cf.fetchAndSetEnumCases(&table)
		if err != nil {
			return err
		}
	}

To Reproduce
Following the examples/ReadMe.md, you will see an error like "Missing primary key" which is caused by this

Expected behavior
the different table should point to different schema object

CUDA 9.2?

I know the documentation states CUDA 9.1 is required, but has there been any testing done with CUDA 9.2? Does that work with any version of Aresdb?

support sql

Is your feature request related to a problem? Please describe.
no

Describe the solution you'd like
support querying AresDB with sql

Describe alternatives you've considered

Additional context

Getting an error while running make test

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:
Taken docker image with cuda 9.1
sudo nvidia-docker pull nvcr.io/nvidia/cuda:9.0-devel-ubuntu16.04
Installed go and glide

curl -O https://storage.googleapis.com/golang/go1.11.2.linux-amd64.tar.gz
tar -xvf go1.11.2.linux-amd64.tar.gz
mv go /usr/local

Setting up the path

export PATH=/usr/local/:$PATH
export GOPATH=/usr/local/
apt-get install glide

Unexpected behavior
go build -o bin/aresd
main.go:18:2: cannot find package "github.com/uber/aresdb/cmd" in any of:
/usr/local/go/src/github.com/uber/aresdb/cmd (from $GOROOT)
/usr/local/src/github.com/uber/aresdb/cmd (from $GOPATH)
Makefile:85: recipe for target 'bin/aresd' failed
make: *** [bin/aresd] Error 1

Desktop (please complete the following information):

OS: Ubuntu 16.04

String support

How to insert rows of string data more than the maximum cardinality (65535) of BigEnum (say 100,000). Is there any way to do that?

Provide sample dataset and queries

Is your feature request related to a problem? Please describe.
Provide sample dataset and queries

Describe the solution you'd like

Describe alternatives you've considered

Additional context

How to delete a table that has already been created

In the documents about aresdb, I can't find any document describes how to delete a table or delete a data in a table. I hope that you can provide a detailed document.

skip metric report for latency lag during recovery redolog replay

Describe the bug
We should skip the latency lag metric report for redolog replay during recovery
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Improve configuration engine

Right now we read config from multiple place: e.g. command line, yaml file and environment.
we may consider use https://github.com/spf13/viper to unify configuration engine

cache dimension table in GPU memory

Is your feature request related to a problem? Please describe.
To avoid repeat dimension table transfer

Describe the solution you'd like
Dimension table is very static, usually got update every several hours, we can lazy load dimension table in GPU and keep there until there is new data update.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Support multiple measures in one query

Is your feature request related to a problem? Please describe.
Right now we only compute one measure in one query however multiple measures are doable with some changes. This should save lots of duplicate computation work on dimension

Describe the solution you'd like
Creating multiple measure vectors and reduce them at one time.

Additional context
Hyperloglog is tricky to be computed together with other agg functions since it does not go through the normal sort reduce stage.

Migrate C++ build targets to Cmake

Is your feature request related to a problem? Please describe.
CMake is the new standard to manage the build of C++ targets.

Describe the solution you'd like
Use a CMakefile to manager C++ targets while still using Makefile for other targets

Additional context
This also gives the benefits of using the c++ github libraries (header only)

uber / aresdb Goto Github PK

aresdb's Introduction

AresDB

Legal Note

Getting started

NVIDIA Driver and CUDA Setup

Environment Variables

Language Requirements

Configure

Local Test

Test Golang Code

Ginkgo

Test C++ Code

google-test

Run AresDB Server

Run AresDB Docker

Documentation

License

aresdb's People

Contributors

Stargazers

Watchers

Forkers

aresdb's Issues

Recommend Projects

Recommend Topics

Recommend Org