chronoxor / cpptrader Goto Github PK

Matching engine benchmark results are missing from README?

I find a lot of CppCommen:: in the source code, but i cann't find the namespace in the project. is it a library? how can I get it? thanks

Hi @chronoxor,

While we wanted to add a market pair with 10 letters, we recently noticed that Symbol name is defined as 8 bytes.

What we thought could be an easy fix (just make char Name[16]) seems to have implication as there are some places where code is optimized with unrolling and bit manipulations.

Before we patch it (and eventually propose it upstream) we wanted to clarify if that kind of change could impact code mechanical sympathy and as we are not high performance cpp expert by any means, wanted you opinion on the matters

Regards

Hi @chronoxor thank You for sharing this great matching engine library.

I would like to ask about AON (All Or None) Order. I'm using an example app "cpptrader-example-matching_engine".
Below is a sequence of order that i enter:

1. Add AON Limit Sell Order : Price = 5; Qty = 100;
2. Add GTC Limit Buy : Price = 99999999; Qty = 50;
3. Add GTC Limit Buy : Price = 5; Qty = 25;
4. Add GTC Limit Buy : Price = 5; Qty = 25;

Now all orders executed, Order Id 2 has executed price = 99999999.
But for Order Id 3 and 4 the executed price are also 99999999.
For Order Id 3 and 4 it should be executed with price = 5 right ?

Below is a screen shot of the example app.

Build command:
cmake.exe --build C:\Users\Asus\Documents\Projects\cpp\CppTrader\cmake-build-debug --target cpptrader-tests -- -j 9

Output error

modules/CppCommon/libcppcommon.a(path.cpp.obj): In function `CppCommon::Path::home()':
C:/Users/Asus/Documents/Projects/cpp/CppTrader/modules/CppCommon/source/filesystem/path.cpp:970: undefined reference to `__imp_GetUserProfileDirectoryW'
C:/Users/Asus/Documents/Projects/cpp/CppTrader/modules/CppCommon/source/filesystem/path.cpp:973: undefined reference to `__imp_GetUserProfileDirectoryW'
modules/CppCommon/libcppcommon.a(uuid.cpp.obj): In function `CppCommon::UUID::Sequential()':
C:/Users/Asus/Documents/Projects/cpp/CppTrader/modules/CppCommon/source/system/uuid.cpp:123: undefined reference to `__imp_UuidCreateSequential'
modules/CppCommon/libcppcommon.a(uuid.cpp.obj): In function `CppCommon::UUID::Random()':
C:/Users/Asus/Documents/Projects/cpp/CppTrader/modules/CppCommon/source/system/uuid.cpp:179: undefined reference to `__imp_UuidCreate'
collect2.exe: error: ld returned 1 exit status
mingw32-make.exe[3]: *** [CMakeFiles\cpptrader-tests.dir\build.make:158: cpptrader-tests.exe] Error 1
mingw32-make.exe[2]: *** [CMakeFiles\Makefile2:498: CMakeFiles/cpptrader-tests.dir/all] Error 2
mingw32-make.exe[1]: *** [CMakeFiles\Makefile2:505: CMakeFiles/cpptrader-tests.dir/rule] Error 2

Cmake output:

-- The C compiler identification is GNU 8.1.0
-- The CXX compiler identification is GNU 8.1.0
-- Check for working C compiler: C:/ProgramData/chocolatey/lib/mingw/tools/install/mingw64/bin/gcc.exe
-- Check for working C compiler: C:/ProgramData/chocolatey/lib/mingw/tools/install/mingw64/bin/gcc.exe - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/ProgramData/chocolatey/lib/mingw/tools/install/mingw64/bin/g++.exe
-- Check for working CXX compiler: C:/ProgramData/chocolatey/lib/mingw/tools/install/mingw64/bin/g++.exe - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- The System identification is Windows-10.0.19041 Windows 10.0.19041 Platform/Windows
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- The System identification is Windows-10.0.19041 Windows 10.0.19041 Platform/Windows
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - done
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- The System identification is Windows-10.0.19041 Windows 10.0.19041 Platform/Windows
-- Could NOT find DbgHelp (missing: DBGHELP_LIBRARY DBGHELP_INCLUDE_DIR) 
-- Could NOT find RPC (missing: RPC_LIBRARY RPC_INCLUDE_DIR) 
-- Could NOT find Userenv (missing: USERENV_LIBRARY USERENV_INCLUDE_DIR) 
-- Found LIBVLD: C:/Users/Asus/Documents/Projects/cpp/CppTrader/modules/CppCommon/modules/vld/vld.lib  
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/Asus/Documents/Projects/cpp/CppTrader/cmake-build-debug

At order.cpp: 83, is the -1000 validation number exactly?

Hello,

I just cloned the project and followed the steps to build the project. I'm on macOS and run the command ./unix.sh inside the build folder which produces an error: ./01-generate.sh: line 6: cmake: command not found

Any ideas on what could be going on?

Thanks

gcc complains when including the bfd.h header:

In file included from /home/winterreise/Github/CppTrader/modules/CppCommon/source/system/stack_trace.cpp:21:
/usr/include/bfd.h:35:2: error: #error config.h must be included before this header
   35 | #error config.h must be included before this header
      |  ^~~~~
make[2]: *** [modules/CppCommon/CMakeFiles/cppcommon.dir/build.make:433: modules/CppCommon/CMakeFiles/cppcommon.dir/source/system/stack_trace.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:663: modules/CppCommon/CMakeFiles/cppcommon.dir/all] Error 2
make: *** [Makefile:160: all] Error 2

Is there support for pegged order types ? or support for limit orders that dynamically move with the best market quotes ?

I am consistently getting a segfault after some millions of calls to AddOrder:

Thread 6 "my_server" received signal SIGSEGV, Segmentation fault.
(gdb) backtrace
#0  0x00007fff803c9b2c in std::__1::pair<unsigned long, CppTrader::Matching::OrderNode*>::pair<unsigned long&, CppTrader::Matching::OrderNode*&, false>(unsigned long&, CppTrader::Matching::OrderNode*&) ()
#1  0x00007fff803bad6e in std::__1::pair<std::__1::__unwrap_ref_decay<unsigned long&>::type, std::__1::__$nwrap_ref_decay<CppTrader::Matching::OrderNode*&>::type> std::__1::make_pair<unsigned long&, CppTrader::M$tching::OrderNode*&>(unsigned long&, CppTrader::Matching::OrderNode*&) ()
#2  0x00007fff803b74c3 in CppTrader::Matching::MarketManager::AddLimitOrder(CppTrader::Matching::Order co$st&, bool) ()
#3  0x00007fff803b6bad in CppTrader::Matching::MarketManager::AddOrder(CppTrader::Matching::Order const&)
    ()

It looks like it is this call.

I have many threads that are receiving orders, creating CppTrader::Matching::Order objects, and placing them into a FIFO concurrent queue: orders_queue.enqueue(order). However, reads from the queue and calls to AddOrder are only ever done by a single thread:

         Order order;
          while (orders_queue.try_dequeue(order)) { // Empty the queue
              if (std::this_thread::get_id() != this_thread_id) {
                  abort();
              }
              market.AddOrder(order);
        }

Something strange I notice is that the segfault occurs at about the same point in execution on every run. I print some order book stats every 5 seconds, and it's always around the same number of orders processed before segfault:

// Run 1:
Matching Engine Stats:
    Order Queue Depth: 0
    Total Orders Count: 2027629
    Total Executed Orders Count: 3170152
    Orders Since Last Stats: 86721
    OrderBook(Symbol=Symbol(Id=1; Name="MYSYMBOL"); Bids=237; Asks=240; BuyStop=0; SellStop=0; TrailingBuyStop=0; TrailingSellStop=0)

// Run 2:
Matching Engine Stats:
    Order Queue Depth: 0
    Total Orders Count: 2070242
    Total Executed Orders Count: 3236818
    Orders Since Last Stats: 88030
    OrderBook(Symbol=Symbol(Id=1; Name="MYSYMBOL"); Bids=257; Asks=238; BuyStop=0; SellStop=0; TrailingBuyStop=0; TrailingSellStop=0)

// Run 3:
Matching Engine Stats:
    Order Queue Depth: 0
    Total Orders Count: 2048591
    Total Executed Orders Count: 3202352
    Orders Since Last Stats: 87628
    OrderBook(Symbol=Symbol(Id=1; Name="MYSYMBOL"); Bids=254; Asks=239; BuyStop=0; SellStop=0; TrailingBuyStop=0; TrailingSellStop=0)

All orders are either buy or sell limit orders. The price and quantity values are randomly generated ints from 1 to 1000.

Do you have any recommendations on how I might be incorrectly handling the memory management of my Orders, or other ideas on what might cause this?

Hi,

I just installed the code and was giving it a try but got the "killed" message when trying the optimized version of the market_manager (the non_optimized version runs without a problem).

Any idea of what is the problem?

Cheers

PS: all tests ran successfully.

cd build
vs.bat
i can not find where vs.bat is. should you help me

Maybe we need cancel the order function.

Hi I would like to ask, how to copy dir CppTrader to different location ? eg : I have done some modification of CppTrader in host PC & was run unix.sh at host PC. but now i want to copy into docker image & compile my code + library CppTrader inside docker build.

due to it has symbolic link, the link become invalid when copied to different path.
Currently my approach is copy modified CppTrader to /tmp -> then git init -> git clone from /tmp to src/libraries/CppTrader -> gil update -> unix.sh again. But this method require additional time to re clone all submodules.

Is there any best way to do it ?

Thank You very much

Hi Ivan

Can someone leave better documentations in each classes, structs and methods. There are those of us who have trouble reading code and want to pursue the black box approach.

Hi, I am interested in using CppTrader for a crypto exchange that I am building. Current price and quantity types, uint64_t, is largely sufficient. But for additional safety, I would like to enable __uint128_t. I modified your CppTrader to support this. The slowdown is marginal. Would you be interested in taking a look at my changes? Or you would add the support yourself? Or it's not a priority for you? The work is quite straightforward.

The following results were obtained from running
cpptrader-performance-matching_engine on 20190730.BX_ITCH_50.

Using 64-bit prices and quantities.ITCH processing...Done!

Errors: 0

Processing time: 3.853 s
Total ITCH messages: 28734686
ITCH message latency: 134 ns
ITCH message throughput: 7456980 msg/s
Total market updates: 51832310
Market update latency: 74 ns
Market update throughput: 13451079 upd/s

Using 128-bit prices and quantities.ITCH processing...Done!

Errors: 0

Processing time: 4.413 s
Total ITCH messages: 28734686
ITCH message latency: 153 ns
ITCH message throughput: 6510991 msg/s
Total market updates: 51832310
Market update latency: 85 ns
Market update throughput: 11744680 upd/s

gcc complains when including the bfd.h header:

In file included from /home/winterreise/Github/CppTrader/modules/CppCommon/source/system/stack_trace.cpp:21:
/usr/include/bfd.h:35:2: error: #error config.h must be included before this header
   35 | #error config.h must be included before this header
      |  ^~~~~
make[2]: *** [modules/CppCommon/CMakeFiles/cppcommon.dir/build.make:433: modules/CppCommon/CMakeFiles/cppcommon.dir/source/system/stack_trace.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:663: modules/CppCommon/CMakeFiles/cppcommon.dir/all] Error 2
make: *** [Makefile:160: all] Error 2

Hi chronoxor,

Thank you for the great order book implementation. Recently I have been playing with it to compare different approaches to order book creations with NYSE Arca data. Your approach was generic, so I have been tempted to adapt it.
At this moment I've been mostly interested in having L1 updates (specifically top of the book for each instrument). Implemented auxilliary structure to track if each update price changed worked just fine, e.g.:

`
OrderBook* orderBookPtr = (OrderBook*)market.GetOrderBook(instrument_id);

uint64_t orig_best_ask_price = 0;
uint64_t orig_best_ask_vol = 0;
if (orderBookPtr->best_ask())
{
orig_best_ask_price = orderBookPtr->best_ask()->Price;
orig_best_ask_vol = orderbookPtr->best_ask()->TotalVolume;
}
// Perform order book calculation e.g.
market.AddOrder(Order::Limit(1234567890, GetInstId("WMT"), OrderSide::SELL, 120000, 25);

if (orig_best_ask_price != orderBookPtr->best_ask()->Price || orig_best_ask_vol != orderBookPtr->best_ask()->TotalVolume)
{
// Update top of book
}
`
The code above works perfectly at least for the following updates: 'Modify', 'Execute', 'Delete', 'Add', and 'Replace'. However, it introduces latency, each check is required if the top of the book has been changed.

Investigating code by examples, I've been intrigued, if MarketHandler interface could be used to make code faster - https://github.com/chronoxor/CppTrader/blob/master/include/trader/matching/market_handler.h

However testing code, looks like 'UpdateLevel'

Is this expected?
I have made some modifications for this code to work for L1 top of book updates and the moment still under testing (and at this moment isn't best effort implementation). But I was more wondering, if I missing a point here what is meant by 'LevelUpdate' member 'top', is it for multiple level depth? It's not clear though, how to know which LevelUpdate 'top' is relating to during callback... Or is it not expected & simply a bug?

To replicate a problem (currently only using NYSE data):

`
#ifndef NYSEFH_SRC_ORDERBOOK_MARKET_HANDLER_EXAMPLE
#define NYSEFH_SRC_ORDERBOOK_MARKET_HANDLER_EXAMPLE

#include <unordered_map>
#include <string>
#include "trader/matching/market_manager.h"

using namespace CppTrader::Matching;

class NYSEMarketHandler: public MarketHandler
{
public:
void onUpdateOrderBook(const OrderBook& orderBook, bool isTop) override
{
if (isTop)
{
static int num{ 0 };
printf("top of the book update! %i\n", num++);
}
}
};

NYSEMarketHandler marketHandler;
MarketManager market(marketHandler);

int main()
{
printf("Start cpptrader testing\n");
std::unordered_map<std::string, uint32_t> instKey {
{ "WMT", 1 }
};

Symbol sym{ instKey["WMT"], "WMT" };
market.AddSymbol(sym);
market.AddOrderBook(sym);
printf("in here#1\n");
market.AddOrder(Order::Limit(1234567890, instKey["WMT"], OrderSide::SELL, 23000, 15));
printf("in here#2\n");
market.AddOrder(Order::Limit(1234567891, instKey["WMT"], OrderSide::SELL, 25000, 25));
printf("in here#3\n");
market.ExecuteOrder(1234567891, 25);
printf("in here#4\n");
printf("Finish cpptrader testing\n");
return EXIT_SUCCESS;

}

#endif
`

Output:

Start cpptrader testing
in here#1
top of the book update! 0
in here#2
in here#3
top of the book update! 1
in here#4
Finish cpptrader testing

Expected output:

Start cpptrader testing
in here#1
top of the book update! 0
in here#2
in here#3
in here#4
Finish cpptrader testing

Thanks

/CppTrader/modules/CppBenchmark/modules/HdrHistogram/src/hdr_histogram_log.c:36:9:
error:
unknown pragma ignored [-Werror,-Wunknown-pragmas]
#pragma ide diagnostic ignored "readability-redundant-declaration"

Dear @chronoxor

i'm using cpptrader-example-matching_engine from commit 43b8cfbc438373d3fb9b784025f6018f453e4e72

I have a Trailing Stop Limit Scenario as shown below :

add symbol 1 BTC_ETH
add book 1
enable matching

add limit sell 1 1 1000 100
add limit sell 2 1 1100 100
add limit sell 3 1 1100 100
add limit buy 4 1 900 100
add trailing stop-limit sell 5 1 700 700 100 100 0
add limit buy 6 1 1000 100
add limit buy 7 1 1100 100
add limit sell 8 1 900 100
add limit buy 9 1 1100 200

At first attempt the Trailing Stop is working and Executed. But When I retry with same scenario (without re-running the executable) the Trailing Stop Limit is not working. And why SellStop value is so big ?:

add limit sell 1 1 1000 100
add limit sell 2 1 1100 100
add limit sell 3 1 1100 100
add limit buy 4 1 900 100
add trailing stop-limit sell 5 1 700 700 100 100 0
add limit buy 6 1 1000 100
add limit buy 7 1 1100 100
add limit sell 8 1 900 100
add limit buy 9 1 1100 200

I will attach the screen shot also to make it easier to see / debug

1. First Attempt and Trailing Stop is working
   

2. Second Attempt with same scenario Trailing Stop is not working (without rerunning the executable)
   

Thank You very much
Have a nice day

Best Regards

Hi
I am trying to port CppTrader as library in Cython 3.0.
I have many problems to build this lib in Cython.
Is there any resources for build in Cython?

===============================================================================
test cases: 18 | 16 passed | 2 failed
assertions: 151 | 149 passed | 2 failed

Hello and thanks for putting together this great repository :). I noticed while reading https://github.com/chronoxor/CppTrader/blob/master/performance/market_manager_optimized_aggressive.cpp that many portions are substantially similar to an implementation I published, https://github.com/charles-cooper/itch-order-book/blob/master/order_book.h, but with no attribution. I recognize that you have licenses and copyrights on your own code, and I'm sure you would like others to respect those, so please do your part in upholding the tenets of the open source community and follow the terms of my license https://github.com/charles-cooper/itch-order-book/blob/master/LICENSE :).

Hi,
I'm new to C++ trading system. Found this great project for learning.
When I tried to build and run the sample benchmark, I found that I can't access to the ftp://emi.nasdaq.com/ITCH to get the ITCH sample data. After spending some time on this, I finally found out https://emi.nasdaq.com/ITCH is a replacement.
Maybe this could help, for someone who also need the sample data.

thank you.

Dear @chronoxor

I would like to ask :

1. What is the best way to get status of Taker or Maker order at callback "onExecutedOrder" ?
   In my case, i need to know which order is Taker / Maker.
2. What is the best way to get the Match Number of the Executed Orders, so we can know which order is matching with the other order.

Thank You very much for Your Great repo
Have a nice day

Best Regards

Rendy

The market_manager_optimized version of the binary segfaults with 01302020.NASDAQ_ITCH50

Hi,

I'm building a simulator using your matching engine,

I was wondering if you have any suggestions on an efficient way to extract a time series (or just an event-by-event vector) of transacted (matched) price and quantity pairs ? In particular, when the matching is enabled at all time.

Best,

Are newer C++11 to C++17 standard features used in CppTrader? What is the minimum version?

Hello @chronoxor

I want to know why the price and quantity don't have a decimal value, is there any particular concern? is there any problem if I change it to decimal value?

Thank you in advance.

I am not that expert in c++, so looking for example to build limit order book from ITCH file for particular ticker. It will make jobs of other people much easier.

Is there commissioned work or paid support for this open source product?