hot-stuff / libhotstuff Goto Github PK

A general-purpose BFT state machine replication library with modularity and simplicity, suitable for building hybrid consensus cryptocurrencies.

License: Apache License 2.0

CMake 2.66% C++ 87.21% Python 7.78% Shell 2.35%

libhotstuff's Introduction

libhotstuff

libhotstuff is a general-purpose BFT state machine replication library with modularity and simplicity, suitable for building hybrid consensus cryptocurrencies.

Paper

This repository includes the prototype implementation evaluated in our HotStuff: BFT Consensus in the Lens of Blockchain paper. The consensus protocol is also used by Facebook in Libra (now rebranded to "Diem") project.

Feel free to contact us if you'd like to reproduce the results in the paper, or tweak the code for your own project/product.

Full paper: https://arxiv.org/abs/1803.05069
PODC 2019 paper: https://dl.acm.org/citation.cfm?id=3331591

Heads Up

Although we don't know why, many developers seem to be stuck with those intermediate algorithms in the paper (e.g. Basic HotStuff) that are only for theory/research purpose. As already noted in the paper, we recommend the last algorithm ("Event-Driven HotStuff") as it is simple, elegant (there is no "view" at all in the core protocol and only one generic type of QCs) and very close to an actual implementation. It is also the original algorithm we came up with, before adding in Basic/Chained HotStuff that may help some researchers understand its theoretical relation to PBFT, etc.

TL;DR: read the spec (and the proof) of the algorithm in "Implementation" section, if you're only curious about using HotStuff in your system/work.

Features

Simplicity. The protocol core logic implementation is as simple as the one specified in our paper. See consensus.h and consensus.cpp.
Modular design. You can use abstraction from the lowest level (the core protocol logic) to the highest level (the state machine replication service with network implementation) in your application, or override/redefine the detailed behavior to customize your own consensus.
Liveness decoupled from safety. The liveness logic is entirely decoupled from safety. By defining your own liveness gadget ("PaceMaker"), you can implement your own liveness heuristics/algorithm. The actual performance varies depending on your liveness implementation, but the safety is always intact.
Friendly to blockchain systems. A PaceMaker could potentially be PoW-based and it makes it easier to build a hybrid consensus system, or use it at the core of some cryptocurrencies.
Minimal. The project strives to keep code base small and implement just the basic functionality of state machine replication: to deliver a consistent command sequence to the library user. Application-specific parts are not included, but demonstrated in the demo program.

Try the Current Version

NOTICE: the project is still in-progress. Try at your own risk, and this section may be incomplete and subject to changes.

# install from the repo
git clone https://github.com/hot-stuff/libhotstuff.git
cd libhotstuff/
git submodule update --init --recursive

# ensure openssl and libevent are installed on your machine, more
# specifically, you need:
#
# CMake >= 3.9 (cmake)
# C++14 (g++)
# libuv >= 1.10.0 (libuv1-dev)
# openssl >= 1.1.0 (libssl-dev)
#
# on Ubuntu: sudo apt-get install libssl-dev libuv1-dev cmake make

cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED=ON -DHOTSTUFF_PROTO_LOG=ON
make

# start 4 demo replicas with scripts/run_demo.sh
# then, start the demo client with scripts/run_demo_client.sh


# Fault tolerance:
# Try to run the replicas as in run_demo.sh first and then run_demo_client.sh.
# Use Ctrl-C to terminate the proposing replica (e.g. replica 0). Leader
# rotation will be scheduled. Try to kill and run run_demo_client.sh again, new
# commands should still get through (be replicated) once the new leader becomes
# stable. Or try the following script:
# scripts/faulty_leader_demo.sh

Try to Reproduce Our Basic Results

See here.

TODO (When I get some free time...)

Rewrite this minimal code base in Rust: this time, with all the experience, without C++ template kung-fu, I plan to have a ready-to-use, blackbox-like libhotstuff implementation as a full library with better encapsulation and interface. The new goal would be any engineer without knowledge of BFT should be able to use it for his/her own application, without changing the library code. Ping me if you like this re-writing idea or you'd like to be part of it.
Limit the async event callback depth (otherwise in the demo a fresh replica could overflow its callback stack when trying to catch up)
Add a PoW-based Pacemaker example
Branch pruning & swapping (the current implementation stores the entire chain in memory)
Persistent protocol state (recovery?)

libhotstuff's People

Contributors

Stargazers

Watchers

libhotstuff's Issues

[Question] Can a malicious proposer cause an infinite non-deciding scenario in 'Basic HotStuff'

The paper BFT Consensus in the Lens of Blockchain describes an infinite non-deciding scenario for a theoretical “two-phase” Hotstuff variant. Specifically, the claim is that the addition of a "precursor" voting round ensures that (n-f) participants know of any 'hidden/missing Lock' and a correct leader would be able to propose a value that all honest participants would accept without waiting the maximum network delay (such as Tendermint / Casper FFG).

During this precursor round, if a quorum is achieved and a certificate is created, Replicas will store the highest Precursor (PrepareQC) they know of in a local variable known as HighQC.

The paper details a 'SafeNodePredicate' Replicas use to determine whether or not to accept a Proposal from a Leader.

However, within the SafeNodePredicate the HighQC (highest Precursor) of any Replica is never checked against the proposal. Could a malicious Proposer simply choose to ignore a HighQC removing the safety properties of the Precursor round and forcing a 'Hidden Lock'?

Chained HotStuff

I'm currently wondering, is the default implementation using the chained mode?
Because I'm seeing a block being committed on every quorum.

[Question]prop.proposer in vote

void HotStuffCore::on_receive_proposal(const Proposal &prop) {
    LOG_PROTO("got %s", std::string(prop).c_str());
    block_t bnew = prop.blk;
    sanity_check_delivered(bnew);
    update(bnew);
    bool opinion = false;
    if (bnew->height > vheight)
    {
        if (bnew->qc_ref && bnew->qc_ref->height > b_lock->height)
        {
            opinion = true; // liveness condition
            vheight = bnew->height;
        }
        else
        {   // safety condition (extend the locked branch)
            block_t b;
            for (b = bnew;
                b->height > b_lock->height;
                b = b->parents[0]);
            if (b == b_lock) /* on the same branch */
            {
                opinion = true;
                vheight = bnew->height;
            }
        }
    }
    LOG_PROTO("now state: %s", std::string(*this).c_str());
    if (bnew->qc_ref)
        on_qc_finish(bnew->qc_ref);
    on_receive_proposal_(prop);
    if (opinion && !vote_disabled)
        do_vote(prop.proposer,
            Vote(id, bnew->get_hash(),
                create_part_cert(*priv_key, bnew->get_hash()), this));
}

As described in paper, in the on_receive_proposal function replica could send voteMsg to a leader which decided by pacemaker getLeader function. However in this code it only pass the proposer of the proposal, am I missing anything here?

about deploy

when I reproduce your basic results, I used four EC2 with Ubuntu 18.04. But when I run ./run.sh new myrun1, there is a mistake said # start the hotstuff replica **********************************************************************************************

replica2 - FAILED!!! ------------------------------------------------------
Unable to import hotstuff_app due to invalid syntax
replica1 - FAILED!!! ------------------------------------------------------
Unable to import hotstuff_app due to invalid syntax
replica0 - FAILED!!! ------------------------------------------------------
Unable to import hotstuff_app due to invalid syntax
.

start the hotstuff replica **********************************************************************************************

replica3 - FAILED!!! ------------------------------------------------------
Unable to import hotstuff_app due to invalid syntax

STATS *******************************************************************************************************************

localhost : ok=7 changed=0 failed=0 unreachable=0 rescued=0 ignored=0
replica0 : ok=5 changed=4 failed=1 unreachable=0 rescued=0 ignored=0
replica1 : ok=5 changed=4 failed=1 unreachable=0 rescued=0 ignored=0
replica2 : ok=5 changed=4 failed=1 unreachable=0 rescued=0 ignored=0
replica3 : ok=5 changed=4 failed=1 unreachable=0 rescued=0 ignored=0

So what's going wrong?

Validation of Quorum Certificates

Hello,

I've been considering using HotStuff in a research project (which means reimplementing parts of the codebase) and had one question about this implementation. When must quorum certificates be checked?

My understanding from reading the HotStuff paper was that a proposal's QC should point to a previous block (hash or block.justify), and that the associated signatures need to be on the previous hash and need to be valid. It seemed as though this means that QCs should be validated upon receipt of a new block proposal.

The Diem consensus implementation appears to follow this behavior, i.e. verify the QC immediately upon receipt of a block proposal.

This implementation, however, appears to not verify the QCs of new proposals. Particularly, I do not see a call site for QuorumSet::verify other than in Block::verify, and I do not see a call to Block::verify other than in HotStuffBase::async_deliver_block (hotstuff.cpp line 188), where the result of checking the QC on a newly received block appears to be used to print a warning and is then discarded.

        std::vector<promise_t> pms;
        const auto &qc = blk->get_qc();
        assert(qc);
        if (blk == get_genesis())
            pms.push_back(promise_t([](promise_t &pm){ pm.resolve(true); }));
        else
            pms.push_back(blk->verify(this, vpool));
        pms.push_back(async_fetch_blk(qc->get_obj_hash(), &replica));
        /* the parents should be delivered */
        for (const auto &phash: blk->get_parent_hashes())
            pms.push_back(async_deliver_blk(phash, replica));
        promise::all(pms).then([this, blk](const promise::values_t values) {
            auto ret = promise::any_cast<bool>(values[0]) && this->on_deliver_blk(blk);
            if (!ret)
                HOTSTUFF_LOG_WARN("verification failed during async delivery");
        });

This would seem to allow delivery of blocks with invalid QCs.

Am I misunderstanding the codebase, or am I misunderstanding the HotStuff paper?

Thanks!

Proposer is not verified

I recently discovered that honest nodes may vote to a proposal that is not the current proposer, which is decided by the pacemaker. While this is fine when all nodes are honest, an attacker can always send inconsistent (but valid) proposals to other nodes so that the other nodes can not agree on a single block, and thus this attacker can DoS the protocol as long as its proposals arrive faster than others' proposals.

I also noticed that in Basic and Chained HotStuff, the pseudo code requires a replica to match message from leader, but this procedure is gone for the pseudo code of Event-driven HotStuff. Maybe there is an assumption that onReceiveProposal will only be called when the proposal is sent from the correct proposer? Or this is just a bug in the pseudo code?

Roundrobin pacemaker not working - Not always rotating

Hi, I ran hotstuff with the pacemaker rr, but the leader position stayed at position 0 over the entire livecycle.
Is there anything else that has to be configured for this?

[question] vheight is never updated

vheight matters for safety, but it's never updated, so I doubt that safety is violated

build error on CentOS 7

Env:
CentOS Linux release 7.6.1810 (Core)
cmake 3.11.0-1.el7
gcc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2)
openssl 1.1.1l

cmake with the following error @Determinant

[apps@mwpl71957 libhotstuff]$ cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED=ON -DHOTSTUFF_PROTO_LOG=ON
-- Found OpenSSL: /usr/lib64/libcrypto.so (found suitable version "1.1.1l", minimum required is "1.1.0") 
-- Found OpenSSL: /usr/lib64/libcrypto.so (found version "1.1.1l") 
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.5") found components:  doxygen missing components:  dot
-- Configuring done
CMake Error in CMakeLists.txt:
  Target "hotstuff" requires the language dialect "CXX17" (with compiler
  extensions), but CMake does not know the compile flags to use to enable it.


CMake Error in CMakeLists.txt:
  Target "hotstuff_shared" requires the language dialect "CXX17" (with
  compiler extensions), but CMake does not know the compile flags to use to
  enable it.


CMake Error in CMakeLists.txt:
  Target "hotstuff_static" requires the language dialect "CXX17" (with
  compiler extensions), but CMake does not know the compile flags to use to
  enable it.


CMake Error in CMakeLists.txt:
  Target "hotstuff-tls-keygen" requires the language dialect "CXX17" (with
  compiler extensions), but CMake does not know the compile flags to use to
  enable it.


CMake Error in CMakeLists.txt:
  Target "hotstuff-keygen" requires the language dialect "CXX17" (with
  compiler extensions), but CMake does not know the compile flags to use to
  enable it.


CMake Error in test/CMakeLists.txt:
  Target "test_secp256k1" requires the language dialect "CXX17" (with
  compiler extensions), but CMake does not know the compile flags to use to
  enable it.


CMake Error in examples/CMakeLists.txt:
  Target "hotstuff-client" requires the language dialect "CXX17" (with
  compiler extensions), but CMake does not know the compile flags to use to
  enable it.


CMake Error in examples/CMakeLists.txt:
  Target "hotstuff-app" requires the language dialect "CXX17" (with compiler
  extensions), but CMake does not know the compile flags to use to enable it.


-- Generating done
-- Build files have been written to: /apps/tmp/libhotstuff

build error

# install from the repo
git clone https://github.com/hot-stuff/libhotstuff.git
cd libhotstuff/
git submodule update --init --recursive

先编译子模块secp256k1

$ ./autogen.sh
$ ./configure --disable-openssl-tests
$ make
$ make check
$ sudo make install  # optional

修改CMakList.txt

set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -W -Wall -Wextra -pedantic -Wsuggest-override -no-pie -fno-pie -fPIC")

# ensure openssl and libevent are installed on your machine, more
# specifically, you need:
#
# CMake >= 3.9 (cmake)
# C++14 (g++)
# libuv >= 1.10.0 (libuv1-dev)
# openssl >= 1.1.0 (libssl-dev)
#
# on Ubuntu: sudo apt-get install libssl-dev libuv1-dev cmake make

# cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED=ON -DHOTSTUFF_PROTO_LOG=ON
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED=OFF -DHOTSTUFF_PROTO_LOG=ON
make

出现如下错误，我该怎么做

Threshold Signatures

Hi,

I've been digging into the code for the past week but I couldn't find if you're actually using the threshold signature scheme.

I couldn't find the distribution of the partial signatures and the methods to compute it is actually empty:

void compute() override {}

Am I missing something?

Run examples on single machines with multiple ips

Hello, I want to run experiments on my single local machine with multiple ips. I defined 4 ips:
192.168.2.14
192.168.2.15
192.168.2.16
192.168.2.17

14 and 15 are my clients and 16, 17 are my replicas. With notice to the link https://github.com/hot-stuff/libhotstuff/tree/master/scripts/deploy I first defined clients.txt like this:

192.168.2.14
192.168.2.15

and replicas.txt like:

192.168.2.16 192.168.2.17
192.168.2.17 192.168.2.16

After running ./gen_all.sh 400 and calling ./run.sh setup I received this error:

# Gathering Facts *********************************************************************************************************
  * 192.168.2.14               - FAILED!!! ------------------------------------------------------
    Failed to connect to the host via ssh: Warning: Permanently added '192.168.2.14' (ED25519) to the list of known hosts.
    [email protected]: Permission denied (publickey,password).                                                               
  * 192.168.2.15               - FAILED!!! ------------------------------------------------------
    Failed to connect to the host via ssh: Warning: Permanently added '192.168.2.15' (ED25519) to the list of known hosts.
    [email protected]: Permission denied (publickey,password).                                                               
  * 192.168.2.16               - FAILED!!! ------------------------------------------------------
    Failed to connect to the host via ssh: Warning: Permanently added '192.168.2.16' (ED25519) to the list of known hosts.
    [email protected]: Permission denied (publickey,password).                                                               
  * 192.168.2.17               - FAILED!!! ------------------------------------------------------
    Failed to connect to the host via ssh: Warning: Permanently added '192.168.2.17' (ED25519) to the list of known hosts.
    [email protected]: Permission denied (publickey,password).                                                               

# STATS *******************************************************************************************************************
192.168.2.14    : ok=0  changed=0       failed=0        unreachable=1   rescued=0       ignored=0
192.168.2.15    : ok=0  changed=0       failed=0        unreachable=1   rescued=0       ignored=0
192.168.2.16    : ok=0  changed=0       failed=0        unreachable=1   rescued=0       ignored=0
192.168.2.17    : ok=0  changed=0       failed=0        unreachable=1   rescued=0       ignored=0

What is the problem? Is it possible to run experiments on a single machine?

How can i test hotstuff in this repo in public network.

I simply change the local IP in the second colomn of the replicas.txt to the server's enternal ip, which is same to the first colomn.
something like this:

# replicas.txt
18.181.230.100      18.181.230.100
35.74.81.208        35.74.81.208
18.183.43.140       18.183.43.140
35.74.78.35         35.74.78.35

but it seems doesn't work.
the client doesn't make progress with no more output.

Failure on large deployments.

It seems that running this with > 250 processes the communication library starts breaking apart badly. It seems to fail to initiate all necessary connections and then errors out on the stats message.
Even wrapping this in a catch doesn't allow me to run it.

libuv的版本问题

请作者在README注明一下依赖的库还需要libuv，以及具体的版本。因为我在跑代码的时候，发现salticidae所注明的libuv >= 1.10.0完全不行，如果恰好使用版本为1.10.0的libuv，那么会在编译时报错。

block-size doesn't work

Hi Ted Yin,
I run the demo with (block-size=1, enable HOTSTUFF_ENABLE_BENCHMARK), and it works well:

# run server
$ ./scripts/run_demo.sh
# run client
$ ./scripts/run_demo_client.sh > hs1.txt 2>&1
# run thr_hist.py
$ cat hs1.txt | python3 ./scripts/thr_hist.py 

# thr_hist.py output
[437, 420, 485, 458, 479, 493, 468, 469, 504, 491, 420, 439, 489, 483, 462, 
475, 456, 460, 513, 515, 231]
lat = 8.476ms
lat = 8.201ms

But when the block-size set to 2, it comes:

$ ./scripts/run_demo.sh

$ ./scripts/run_demo_client.sh > hs2.txt 2>&1

$ cat hs2.txt | python3 ./scripts/thr_hist.py 

# thr_hist.py output
[4, 4, 4, 4, 4, 4, 4, 4, 4, 4]
lat = 11658.689ms
lat = 12006.291ms

It falls down dramatically, and the stranger thing is yet to come:
I set the block-size to 8000, it comes:

$ ./scripts/run_demo.sh

$ ./scripts/run_demo_client.sh > hs8000.txt 2>&1

$ cat hs8000.txt | python3 ./scripts/thr_hist.py 

# thr_hist.py output
[4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4]
lat = 11787.811ms
lat = 12012.320ms

I didn't figure out why is this happend.

Build error on mac

warning: unknown warning option '-Wsuggest-override'; did you mean '-Wshift-overflow'? [-Wunknown-warning-option]
In file included from /Users/kaichen/Documents/projects/libhotstuff/salticidae/src/event.cpp:3:
/Users/kaichen/Documents/projects/libhotstuff/salticidae/include/salticidae/event.h:31:10: fatal error: 'uv.h' file
not found
#include <uv.h>
^~~~~~
1 warning and 1 error generated.
make[2]: *** [salticidae/CMakeFiles/salticidae.dir/src/event.cpp.o] Error 1
make[1]: *** [salticidae/CMakeFiles/salticidae.dir/all] Error 2
make: *** [all] Error 2

[Question]No found newView method in pacemaker

After read paper, there should be two newView related method, however not found in code. Am I missing anything?

Failed to run the experiment

Hey there! Been playing around with HotStuff a bit and I'm having trouble getting results as described in the "Run the experiment" step. In particular, running the command cat myrun1_cli/remote/*/log/stderr | python ../thr_hist.py gives me a division by zero error on this line because of lats being 0. I think the regex doesn't capture the lines in the stderr logs. My logs look something like this (repeat 10x times):

2022-07-27 19:58:58.301778 [hotstuff info] send new cmd d68a66fa09
2022-07-27 19:58:58.301795 [hotstuff info] send new cmd f2a842a75f
2022-07-27 19:58:58.301814 [hotstuff info] send new cmd 3ef8613879
2022-07-27 19:58:58.301831 [hotstuff info] send new cmd 6dafa0339c
2022-07-27 19:58:58.301848 [hotstuff info] send new cmd 5179dece19
2022-07-27 19:58:58.301865 [hotstuff info] send new cmd b9a9d67a1a
2022-07-27 19:58:58.301883 [hotstuff info] send new cmd 8914378bec
2022-07-27 19:58:58.301899 [hotstuff info] send new cmd decbf7dcf3
2022-07-27 19:58:58.301916 [hotstuff info] send new cmd 73f0626419
2022-07-27 19:58:58.301933 [hotstuff info] send new cmd 1ef4265c47
2022-07-27 19:58:58.301950 [hotstuff info] send new cmd a5258a7b4f

My wild guess is that my config is wrong and replicas don't commit, while the regex looks for commit messages. Would really appreciate if you could confirm this suspicion. Thanks!

OnReceiveProPosal question

I noticed in the latest paper's Event-driven code, Update() called after sending vote. But in libhotstuff code, Update() called before sending vote. I'm confused as Update() may update lock and sending vote depends on lock. So which is correct? Or both can be correct? Thanks.

about implementation

I have a confusion about the implementation.
Since C++11 has implemented Promise features/functions, why do the authors implement a new one in the include/hotstuff/promise.hpp file?

Could anyone help me to clear my confusion? Thanks a lot!

副本节点挂机

在共识中投票是发给下一轮的leader，如果该leader节点挂机了，那么这笔proposal该怎么共识？

Connection Losses when sending high quantities of data shortly after another

I've noticed that when CPU spikes happen a lot of connections break up and I also get strange error messages as:

2021-02-08 22:59:58.014039 [net info] send(89) failure: Success
2021-02-08 22:59:58.014100 [net info] send(89) failure: Success

2021-02-08 22:59:58.014155 [net info] 85f483e8f5: lost connection <Conn fd=89 addr=<NetAddr 10.1.0.57:20001> mode=active>
2021-02-08 22:59:58.014179 [net info] ended 85f483e8f5 <-/-> ae0d049455 (via <Conn fd=89 addr=<NetAddr 10.1.0.57:20001> mode=active>)

It doesn't say anything about the reason of the connection losses either. Is there a way to mitigate that?
It's strange to handle because data gets lost.

Reproducing results locally seems to work, but latency analysis does not

Hi!

Thanks a lot for your detailed documentation on how to run HotStuff and reproduce the results.

I managed to get it running locally; however, I ran into the following error:

cat myrun1_cli/remote/*/log/stderr | python ../thr_hist.py
Traceback (most recent call last):
  File "../thr_hist.py", line 71, in <module>
    print("lat = {:.3f}ms".format(sum(lats) / len(lats) * 1e3))
ZeroDivisionError: division by zero

This is because for some reason the regex ([^[].*) \[hotstuff info\] ([0-9.]*)$ does not match a single line in the logs. Here is a small snippet of what they look like:

2020-11-29 16:37:16.255842 [net info] created <Conn fd=11 addr=<NetAddr 127.0.0.1:20000>
 mode=active>
2020-11-29 16:37:16.255933 [net info] connected to remote <Conn fd=11 addr=<NetAddr 127.
0.0.1:20000> mode=active>
2020-11-29 16:37:16.256231 [net info] created <Conn fd=19 addr=<NetAddr 127.0.0.1:20001>
 mode=active>
2020-11-29 16:37:16.256676 [net info] created <Conn fd=21 addr=<NetAddr 127.0.0.1:20002>
 mode=active>
2020-11-29 16:37:16.256711 [net info] connected to remote <Conn fd=19 addr=<NetAddr 127.
0.0.1:20001> mode=active>
2020-11-29 16:37:16.256725 [net info] connected to remote <Conn fd=21 addr=<NetAddr 127.
0.0.1:20002> mode=active>
2020-11-29 16:37:16.256946 [net info] created <Conn fd=23 addr=<NetAddr 127.0.0.1:20003>
 mode=active>
2020-11-29 16:37:16.256993 [net info] connected to remote <Conn fd=23 addr=<NetAddr 127.
0.0.1:20003> mode=active>
2020-11-29 16:37:16.259388 [hotstuff info] send new cmd 1b9d5b531e
2020-11-29 16:37:16.259400 [hotstuff info] send new cmd 278f57afb8
2020-11-29 16:37:16.259413 [hotstuff info] send new cmd c07f8e1d9d
2020-11-29 16:37:16.259425 [hotstuff info] send new cmd 1e664d3172
2020-11-29 16:37:17.614404 [hotstuff info] got <fin decision=1 cmd_idx=0 cmd_height=1 cmd=af5570f5a1 blk=9d62adae67>, wall: 1.357, cpu: 0.017
2020-11-29 16:37:17.614481 [hotstuff info] send new cmd 0889c974f1
2020-11-29 16:37:17.614514 [hotstuff info] got <fin decision=1 cmd_idx=1 cmd_height=1 cmd=01acecb507 blk=9d62adae67>, wall: 1.357, cpu: 0.017
2020-11-29 16:37:17.614540 [hotstuff info] send new cmd 1bad60c45e
2020-11-29 16:37:17.614567 [hotstuff info] got <fin decision=1 cmd_idx=2 cmd_height=1 cmd=2fcd151b82 blk=9d62adae67>, wall: 1.357, cpu: 0.017
2020-11-29 16:37:17.614593 [hotstuff info] send new cmd f8a200dab9
2020-11-29 16:37:17.614621 [hotstuff info] got <fin decision=1 cmd_idx=3 cmd_height=1 cmd=9ae4c9d086 blk=9d62adae67>, wall: 1.357, cpu: 0.017
2020-11-29 16:37:17.614644 [hotstuff info] send new cmd d1f3be1b5e
2020-11-29 16:37:17.614674 [hotstuff info] got <fin decision=1 cmd_idx=4 cmd_height=1 cmd=1b03ab083d blk=9d62adae67>, wall: 1.358, cpu: 0.017
2020-11-29 16:37:17.614696 [hotstuff info] send new cmd 7762cb05ad

This looked fine to me. I used the default settings and ran the clients on localhost (the replicas as well). I understand that this can lead to sub-optimal performance, but I just want to test things out.
The logs of the replicas look fine too, but they are not used for your latency analysis anyway (as far as I can tell).

Could you please confirm that the regex is indeed correct? Did I miss anything else?
Could you maybe provide a small snippet of what the logs are supposed to look like?

Thanks a lot :)

Payload size

Hi,

I have deployed Hotstuff in AWS successfully, but I did not find a place to set payload size.
I am not sure that it is the right way to define HOTSTUFF_CMD_REQSIZE and HOTSTUFF_CMD_RESPSIZE in client.h. When I set these two values as 1024 bytes, the client seemed unable to send requests to the replicas.