hstreamdb / hstream Goto Github PK

Since we need to provide an admin dashboard.
We will need provide some http apis for stream. But now we can only get StreamNames from HStore by findStreams.
Maybe we should return more detailed information (such as replicationFactor and beginTimestamp) instead of just StreamName .
Also provide another api for query stream info by streamname.

• How to build Haskell projects by docker? ( with some extra shared library, e.g., logdevice, zookeeper
• How to make Haskell-Language-Server work?
• If I want to edit locally and build remotely, show me the steps.

FAQs:

• Q: Why my hls doesn't find hsc2hs but cabal build success?
• A: Short answer: run cabal install hsc2hs

For example:

INSERT INTO foo (a,  b) VALUES (1, "xxx{}");

╭─────┬─────╮
│  a  │  b  │
╞═════╪═════╡
│ 1.0 │ xxx │
╰─────┴─────╯

创建流，ctl 报错

CREATE STREAM demo WITH (FORMAT = "JSON");

HttpExceptionRequest Request {
host = "localhost"
port = 6570
secure = False
requestHeaders = [("Content-Type","application/json; charset=utf-8")]
path = "/create/query"
queryString = ""
method = "POST"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
proxySecureMode = ProxySecureWithConnect
}
(ConnectionFailure Network.Socket.connect: <socket: 12>: does not exist (Connection refused))

HStream-Server 报错
hstream-server: LOGS_SECTION_MISSING {name:LOGS_SECTION_MISSING, description:LOGS_SECTION_MISSING: Configuration file misses logs section, callstack:CallStack (from HasCallStack):
throwStreamErrorIfNotOK', called at ./HStream/Store/Exception.hs:324:28 in hstream-store-0.1.0.0-00fd332cb31e900c02128ac35e4d993df94b7e2480f7781981c70e854378ff8e:HStream.Store.Exception}

• exactly-once
• automatically

To reproduce the bug, run

1. cabal run hstream-server
2. cabal run hstream-client and then SELECT * FROM foo EMIT CHANGES;
3. cabal run hstream-client and then two INSERT INTO foo (a, b) VALUES (1, 2);

Details:

hstream-server: Data.HashMap.Internal.(!): key not found
CallStack (from HasCallStack):
  error, called at ./Data/HashMap/Internal.hs:785:16 in unordered-containers-0.2.14.0-4623fa3c6c98778c0d1a2c886970786fa56dd6d9087fb385e168ce0ef5ba869b:Data.HashMap.Internal
  !, called at src/HStream/Processing/Processor.hs:130:49 in hstream-processing-0.1.0.0-inplace:HStream.Processing.Processor

It means that the processor crashed. Logs say that the srcStream in the second SourceRecord does not match the real one:

2021-06-08 07:34:58.410204: [debug] polled 1 records
@(src/HStream/Processing/Processor.hs:122:11)
2021-06-08 07:34:58.410312: [debug] ---------------------------------
@(src/HStream/Processing/Processor.hs:126:17)
2021-06-08 07:34:58.410348: [debug] srcSource: fool﮿
@(src/HStream/Processing/Processor.hs:127:17)
2021-06-08 07:34:58.410377: [debug] expected: foo
@(src/HStream/Processing/Processor.hs:128:24)
2021-06-08 07:34:58.410406: [debug] ---------------------------------

• display empty info while running show streams; if there are no streams.
• some sql examples (:h)

e.g. https://github.com/hstreamdb/hstream/runs/2408537502

>SELECT * FROM demo WHERE humidity > 70 EMIT CHANGES;
ParseException at <unknown position>: syntax error at line 1, column 40 before `EMIT'
CallStack (from HasCallStack):
  ParseException, called at src/HStream/SQL/Parse.hs:23:36 in hstream-sql-0.1.0.0-5c9dc46b93814cbb293cf4f8ba64a8d7fd1e433bdd707f130f0d94c72e0b6ca1:HStream.SQL.Parse
  parse, called at src/HStream/SQL/Parse.hs:30:24 in hstream-sql-0.1.0.0-5c9dc46b93814cbb293cf4f8ba64a8d7fd1e433bdd707f130f0d94c72e0b6ca1:HStream.SQL.Parse
  parseAndRefine, called at client/Main.hs:99:33 in main:Main

Following the steps in the quick start.

EMIT CHANGES must be capital now, or it will get a parsing error. we should make all SQL keywords case insensitive.

• rename topics. We need a HStream.Store.Stream.Topic.rename function to rename our topics. (Btw, a possible logdevice API is Client.rename.)
• remove topics. (Client.removeLogGroup may help)

Describe the bug

> CREATE SOURCE | SINK CONNECTOR mysqltest22 WITH (type = "mysql", host = "0.0.0.0", port = 3306, streamname = "aaa") ;
Client Error: ClientIOError (GRPCIOBadStatusCode StatusCancelled (StatusDetails {unStatusDetails = "Cancelled"}))

To Reproduce
see above

Expected behavior
display warning: Stream doesn't exist

I couldn't find any documentation on how I can connect with HStreamDB server programmatically. I am looking for Java connector. If the connector does not exist then what is the best way to create one.

checkpointed reader

c api: "logdevice/include/CheckpointedReaderBase.h"

• CheckpointedReaderBase.asyncWriteCheckpoints
• CheckpointedReaderBase.asyncWriteCheckpoints (writeLastCheckpointsSync)

checkpoint store

c api: "logdevice/include/CheckpointStore.h"

• CheckpointStore.getLSN
• CheckpointStore.updateLSN

I follow the Quickstart instructions, when I run this command
docker run -td --rm --name some-hstream-store -v dbdata:/data/store --network host hstreamdb/logdevice ld-dev-cluster --root /data/store --use-tcp ,
then shows
hstream-server: error while loading shared libraries: libgrpc.so.14: cannot open shared object file: No such file or directory
and Start HStreamDB Server command also

• logs show
• logs rename
• logs info

ref: logdevice::Clinet::trim

logs show --path [...] --id [...] --max-depth ... --verbose

Tasks

• Add logDirLogsNames
• Add logDirChildFullName and logDirLogFullName, which will be used to get log groups and sub directories
• Add logDirectoryGetAttrs
• ldLogAttrsPtrToHsAttrs which reads attributes from a foreign pointer to a map in Haskell
• Update optparser

Required but not implemented yet:

• logs update
• logs set-range
• logs trim
• nodes-config edit
• nodes-config shrink
• maintenance
• SQL: show tables;
• SQL: describe table_name;
• SQL: select ...

Other commands in ldshell:

• check-internal-logs

We have a Haskell application and I'm wondering since this is written in Haskell if it's possible to just bind directly into the query system without using SQL?

e.g. https://github.com/hstreamdb/hstream/runs/2770152235

Add a RPC has-topic: topic -> bool

Stop reading by stream name

• getVersion: wrapper for getVersion() Thrift method
• getStatus: wrapper for getStatus() Thrift method
• aliveSince: wrapper for aliveSince() Thrift method
• getPid: wrapper for getPid() Thrift method
• get_nodes_state
• add_nodes
• update_nodes
• remove_nodes
• get_nodes_config
• add_nodes
• update_nodes
• remove_nodes
• mark_shards_as_provisioned
• bump_node_generation
• get_maintenances
• apply_maintenance
• remove_maintenances
• mark_all_shards_unrecoverable
• check_impact
• get_log_tree_info
• get_replication_info
• get_settings
• take_log_tree_snapshot
• take_maintenance_log_snapshot
• get_log_group_throughput
• get_log_group_custom_counters

ref: logdevice::Client::getLogGroupById