bitbouncer / kspp Goto Github PK

Since the logic is as follows "if (!upstream_of_something) "... and you cannot attach anything downstream of a vistor it's wrongly assumed to be a top-of-the graph processor

Hi I am on Ubuntu 18.04 and avrogencpp fails to build during the 3rd party install phase:

if (CMAKE_COMPILER_IS_GNUCXX)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall")
if (AVRO_ADD_PROTECTOR_FLAGS)
    set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -fstack-protector-all -D_GLIBCXX_DEBUG")
    # Unset _GLIBCXX_DEBUG for avrogencpp.cc because using Boost Program Options
    # leads to linking errors when compiling with _GLIBCXX_DEBUG as described on
    # http://stackoverflow.com/questions/19729036/
    set_source_files_properties(impl/avrogencpp.cc PROPERTIES COMPILE_FLAGS "-U_GLIBCXX_DEBUG")
endif ()
endif ()
...
[ 70%] Linking CXX executable avrogencpp
CMakeFiles/avrogencpp.dir/impl/avrogencpp.cc.o: In function `boost::re_detail_107000::perl_matcher<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::unwind_extra_block(bool)':
avrogencpp.cc:(.text._ZN5boost16re_detail_10700012perl_matcherIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEESaINS_9sub_matchISC_EEENS_12regex_traitsIcNS_16cpp_regex_traitsIcEEEEE18unwind_extra_blockEb[_ZN5boost16re_detail_10700012perl_matcherIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEESaINS_9sub_matchISC_EEENS_12regex_traitsIcNS_16cpp_regex_traitsIcEEEEE18unwind_extra_blockEb]+0x22): undefined reference to `boost::re_detail_107000::put_mem_block(void*)'
...

full logfile attached
logfile.txt

hey,

I did the steps specified for installing kspp library and now I want to run an example filter-example. But I faced error #include <kspp/impl/serdes/text_serdes.h>.

Should I do anything more like linking something in compiling or else?

If there are many unsent messages in the elastic search sink when exiting ~topology() hangs

last log:
I0816 17:17:49.539950 18348 topology.cpp:23] topology, name:[kspp::elasticsearch_sink]#0 terminating

Hi all. While I know this isn’t an issue I don’t know where else to ask this.

I am currently developing a Kafka application to stream live stock trading prices in real time. Most of the project is complete but I am having a hard time understanding how to join Kafka streams together.

I have one topic in the application, where each partition in that topic is assigned a stock ticker.

Without going into detail why, what I need to do is to be able to join 2 or 3 Kafka partitions together so I can do real time mathematical computations based off the message values of each partition stream.

I tried looking at some of the examples but I am bit confused if what I am trying to do is possible with this library.

Any help would be appreciated. Thanks.

root@IIB-RND:/home/pvssrikanth/srik/kspp/build# make -j "$(getconf _NPROCESSORS_ONLN)"
[ 3%] Built target kspp_grpc_s
[ 7%] Built target kspp_grpc
[ 29%] Built target kspp
[ 51%] Built target kspp_s
[ 52%] Built target kspp_avrogencpp
[ 53%] Built target csv2avro
[ 54%] Built target kafka2postgres
[ 55%] Built target tds2kafka
[ 56%] Built target example_setup
[ 57%] Built target kafka2rocksdb
[ 58%] Built target postgres2kafka
[ 59%] Built target kafka2influxdb
[ 60%] Built target example2-join
[ 61%] Built target kafka2es
[ 62%] Built target avro2pg
[ 63%] Built target bb2pg
[ 64%] Built target bb2console
[ 65%] Built target kafka2avro
[ 66%] Built target bb2avro
[ 67%] Built target example1-partition-topology
[ 68%] Built target example4-count-large
[ 69%] Built target example3-count
[ 70%] Built target example6-filter
[ 72%] Built target example8-ktable-mem
[ 72%] Built target example7-token-bucket
[ 73%] Built target example5-repartition
[ 74%] Built target test14_async
[ 75%] Built target test1
[ 76%] Built target example10-avro
[ 78%] Built target example9-ktable-rocksdb
[ 78%] Built target aws-kinesis-consumer
[ 79%] Built target example12-generic-avro
[ 80%] Built target example14-raw-sink
[ 81%] Linking CXX executable ../../../bin/aws-iot-sample
[ 82%] Built target test2_rocksdb_windowed_store
[ 82%] Linking CXX executable ../../../bin/mqtt-producer
[ 83%] Built target test2_rocksdb_store
[ 84%] Built target test7_cluster_uri
Scanning dependencies of target test5_kafka_source_sink
Scanning dependencies of target test10_http
Scanning dependencies of target test7_url_vector
Scanning dependencies of target test8_join
Scanning dependencies of target test11_metrics
[ 85%] Building CXX object tests/CMakeFiles/test7_url_vector.dir/test7_url_vector.cpp.o
[ 86%] Built target test2_mem_counter_store
[ 88%] Built target test2_mem_windowed_store
[ 88%] Built target test13_mem
[ 89%] Built target test2_mem_store
[ 90%] Built target test2_rocksdb_counter_store
[ 91%] Built target test3_mem_token_bucket
[ 92%] Built target test12_s3
[ 93%] Built target test6_repartition
[ 94%] Built target test4_kafka_consumer
[ 95%] Building CXX object tests/CMakeFiles/test10_http.dir/test10_http.cpp.o
[ 96%] Building CXX object tests/CMakeFiles/test5_kafka_source_sink.dir/test5_kafka_source_sink.cpp.o
[ 97%] Building CXX object tests/CMakeFiles/test8_join.dir/test8_join.cpp.o
[ 98%] Building CXX object tests/CMakeFiles/test11_metrics.dir/test11_metrics.cpp.o
/usr/bin/ld: ../../../lib/libkspp_s.a(mqtt_producer.cpp.o): undefined reference to symbol 'MQTTAsync_isConnected'
//usr/local/lib/libpaho-mqtt3as.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
examples/mqtt-producer/CMakeFiles/mqtt-producer.dir/build.make:105: recipe for target '../bin/mqtt-producer' failed
make[2]: *** [../bin/mqtt-producer] Error 1
CMakeFiles/Makefile2:2758: recipe for target 'examples/mqtt-producer/CMakeFiles/mqtt-producer.dir/all' failed
make[1]: *** [examples/mqtt-producer/CMakeFiles/mqtt-producer.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: ../../../lib/libkspp_s.a(mqtt_producer.cpp.o): undefined reference to symbol 'MQTTAsync_isConnected'
//usr/local/lib/libpaho-mqtt3as.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
examples/aws-iot-sample/CMakeFiles/aws-iot-sample.dir/build.make:108: recipe for target '../bin/aws-iot-sample' failed
make[2]: *** [../bin/aws-iot-sample] Error 1
CMakeFiles/Makefile2:2702: recipe for target 'examples/aws-iot-sample/CMakeFiles/aws-iot-sample.dir/all' failed
make[1]: *** [examples/aws-iot-sample/CMakeFiles/aws-iot-sample.dir/all] Error 2
[ 98%] Linking CXX executable bin/test7_url_vector
[ 98%] Built target test7_url_vector
[ 98%] Linking CXX executable bin/test10_http
[ 98%] Built target test10_http
[ 98%] Linking CXX executable bin/test5_kafka_source_sink
[ 98%] Linking CXX executable bin/test11_metrics
[ 98%] Built target test5_kafka_source_sink
[ 98%] Built target test11_metrics
[ 99%] Linking CXX executable bin/test8_join
[ 99%] Built target test8_join
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
root@IIB-RND:/home/pvssrikanth/srik/kspp/build#

PLEASE KIND HELP HOW TO FIX THIS ISSUE and mail me at [email protected]

thanks

There have been a lot of changes the latest month.

When will v0.7.8 (or v0.8.0) be released?

At this moment app_info assumes multi instance apps are in different consumer groups, which makes them not scalable.

root@IIB-RND:/home/pvssrikanth/srik/kspp/build# make -j "$(getconf _NPROCESSORS_ONLN)"
[ 3%] Built target kspp_grpc_s
[ 7%] Built target kspp_grpc
[ 29%] Built target kspp
[ 51%] Built target kspp_s
[ 52%] Built target kspp_avrogencpp
[ 53%] Built target csv2avro
[ 54%] Built target kafka2postgres
[ 55%] Built target tds2kafka
[ 56%] Built target example_setup
[ 57%] Built target kafka2rocksdb
[ 58%] Built target postgres2kafka
[ 59%] Built target kafka2influxdb
[ 60%] Built target example2-join
[ 61%] Built target kafka2es
[ 62%] Built target avro2pg
[ 63%] Built target bb2pg
[ 64%] Built target bb2console
[ 65%] Built target kafka2avro
[ 66%] Built target bb2avro
[ 67%] Built target example1-partition-topology
[ 68%] Built target example4-count-large
[ 69%] Built target example3-count
[ 70%] Built target example6-filter
[ 72%] Built target example8-ktable-mem
[ 72%] Built target example7-token-bucket
[ 73%] Built target example5-repartition
[ 74%] Built target test14_async
[ 75%] Built target test1
[ 76%] Built target example10-avro
[ 78%] Built target example9-ktable-rocksdb
[ 78%] Built target aws-kinesis-consumer
[ 79%] Built target example12-generic-avro
[ 80%] Built target example14-raw-sink
[ 81%] Linking CXX executable ../../../bin/aws-iot-sample
[ 82%] Built target test2_rocksdb_windowed_store
[ 82%] Linking CXX executable ../../../bin/mqtt-producer
[ 83%] Built target test2_rocksdb_store
[ 84%] Built target test7_cluster_uri
Scanning dependencies of target test5_kafka_source_sink
Scanning dependencies of target test10_http
Scanning dependencies of target test7_url_vector
Scanning dependencies of target test8_join
Scanning dependencies of target test11_metrics
[ 85%] Building CXX object tests/CMakeFiles/test7_url_vector.dir/test7_url_vector.cpp.o
[ 86%] Built target test2_mem_counter_store
[ 88%] Built target test2_mem_windowed_store
[ 88%] Built target test13_mem
[ 89%] Built target test2_mem_store
[ 90%] Built target test2_rocksdb_counter_store
[ 91%] Built target test3_mem_token_bucket
[ 92%] Built target test12_s3
[ 93%] Built target test6_repartition
[ 94%] Built target test4_kafka_consumer
[ 95%] Building CXX object tests/CMakeFiles/test10_http.dir/test10_http.cpp.o
[ 96%] Building CXX object tests/CMakeFiles/test5_kafka_source_sink.dir/test5_kafka_source_sink.cpp.o
[ 97%] Building CXX object tests/CMakeFiles/test8_join.dir/test8_join.cpp.o
[ 98%] Building CXX object tests/CMakeFiles/test11_metrics.dir/test11_metrics.cpp.o
/usr/bin/ld: ../../../lib/libkspp_s.a(mqtt_producer.cpp.o): undefined reference to symbol 'MQTTAsync_isConnected'
//usr/local/lib/libpaho-mqtt3as.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
examples/mqtt-producer/CMakeFiles/mqtt-producer.dir/build.make:105: recipe for target '../bin/mqtt-producer' failed
make[2]: *** [../bin/mqtt-producer] Error 1
CMakeFiles/Makefile2:2758: recipe for target 'examples/mqtt-producer/CMakeFiles/mqtt-producer.dir/all' failed
make[1]: *** [examples/mqtt-producer/CMakeFiles/mqtt-producer.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: ../../../lib/libkspp_s.a(mqtt_producer.cpp.o): undefined reference to symbol 'MQTTAsync_isConnected'
//usr/local/lib/libpaho-mqtt3as.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
examples/aws-iot-sample/CMakeFiles/aws-iot-sample.dir/build.make:108: recipe for target '../bin/aws-iot-sample' failed
make[2]: *** [../bin/aws-iot-sample] Error 1
CMakeFiles/Makefile2:2702: recipe for target 'examples/aws-iot-sample/CMakeFiles/aws-iot-sample.dir/all' failed
make[1]: *** [examples/aws-iot-sample/CMakeFiles/aws-iot-sample.dir/all] Error 2
[ 98%] Linking CXX executable bin/test7_url_vector
[ 98%] Built target test7_url_vector
[ 98%] Linking CXX executable bin/test10_http
[ 98%] Built target test10_http
[ 98%] Linking CXX executable bin/test5_kafka_source_sink
[ 98%] Linking CXX executable bin/test11_metrics
[ 98%] Built target test5_kafka_source_sink
[ 98%] Built target test11_metrics
[ 99%] Linking CXX executable bin/test8_join
[ 99%] Built target test8_join
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
root@IIB-RND:/home/pvssrikanth/srik/kspp/build#

PLEASE KIND HELP HOW TO FIX THIS ISSUE and mail me at [email protected]

thanks

binary_serdes::encode()/decode() do not take endianess into account and always use current platform one. This can cause incompatibilities when other services are on different platforms.

Currently streams need manual partition assignment, but they should also be able to leverage automatic balanced assignment.

grpc fails to install due to busybox errors

Dear Sir,

I have installed docker file on ubuntu linux and following message is appearing.

dpkg-query: no path found matching pattern librdkafka.so.1
dpkg-query: no path found matching pattern librocksdb.so.5.18
dpkg-query: no path found matching pattern libthrift-0.12.0.so
Removing intermediate container 9d07d030d941
---> 82a49ef44181
[Warning] One or more build-args [IMAGE_TAG] were not consumed
Successfully built 82a49ef44181
Successfully tagged kspp-build-ubuntu:latest
~/srik/stream/kspp/docker-ubuntu
af2b83faeef06c49b22ac455aa5acdc6e756596bdc403b69eb683715187d1860

IS IT RUNNING FINE ?

If so could you please help me how to test it by sending email to [email protected]

Thanks in advance

if connection is closed by server the client reconnects but fails to resend the batch that failed

Hi bitbouncer,
I saw the librdkafka have Full Exactly-Once-Semantics (EOS) support.
And I can not find any transaction code in kspp.
So do you have any plan to involve the feature ?

Dear Sir,
Unable to run docker-ubuntu on ubuntu18 Its failing

pvssrikanth@IIB-RND:/srik/kspp/docker-ubuntu$ sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7a11599b10c6 79d49ec93c0f "/bin/bash" 22 seconds ago Exited (0) 20 seconds ago brave_robinson
6c5f757da6b1 d35dec23be9e "/bin/bash" 38 minutes ago Exited (0) 38 minutes ago inspiring_brattain
264c658a12cb d35dec23be9e "/bin/bash" 38 minutes ago Exited (0) 38 minutes ago epic_kowalevski
pvssrikanth@IIB-RND:/srik/kspp/docker-ubuntu$
pvssrikanth@IIB-RND:/srik/kspp/docker-ubuntu$
pvssrikanth@IIB-RND:/srik/kspp/docker-ubuntu$

Currently kspp only seems to support left joins. It would be great to support other join types, e.g. inner join: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Join+Semantics

Currently topology::flush() will loop forever while there are still messages. This doesn't play nice when an app is using its own event loop, sice flush() can theoretically never return. Adding a max processed event parameter can solve this problem.

Also, the usage is non-obvious in case only sinks are used. Examples show topology created with kspp::kafka_partition_sink never calls start()/flush()/close(), when in other cases those methods are used. It would be great to document how a topology should be handled.

currently one must specify a offset storage provider for non kafka sources - however if one know that were only interested in last commited offsets - no such thing is needed. Add a null provider with empty implementation