chronoxor / cpptrader Goto Github PK
View Code? Open in Web Editor NEWHigh performance components for building Trading Platform such as ultra fast matching engine, order book processor
License: MIT License
High performance components for building Trading Platform such as ultra fast matching engine, order book processor
License: MIT License
Hi,
I'm new to C++ trading system. Found this great project for learning.
When I tried to build and run the sample benchmark, I found that I can't access to the ftp://emi.nasdaq.com/ITCH to get the ITCH sample data. After spending some time on this, I finally found out https://emi.nasdaq.com/ITCH is a replacement.
Maybe this could help, for someone who also need the sample data.
thank you.
Hi, I am interested in using CppTrader for a crypto exchange that I am building. Current price and quantity types, uint64_t, is largely sufficient. But for additional safety, I would like to enable __uint128_t. I modified your CppTrader to support this. The slowdown is marginal. Would you be interested in taking a look at my changes? Or you would add the support yourself? Or it's not a priority for you? The work is quite straightforward.
The following results were obtained from running
cpptrader-performance-matching_engine
on 20190730.BX_ITCH_50
.
Using 64-bit prices and quantities.ITCH processing...Done!
Errors: 0
Processing time: 3.853 s
Total ITCH messages: 28734686
ITCH message latency: 134 ns
ITCH message throughput: 7456980 msg/s
Total market updates: 51832310
Market update latency: 74 ns
Market update throughput: 13451079 upd/s
Using 128-bit prices and quantities.ITCH processing...Done!
Errors: 0
Processing time: 4.413 s
Total ITCH messages: 28734686
ITCH message latency: 153 ns
ITCH message throughput: 6510991 msg/s
Total market updates: 51832310
Market update latency: 85 ns
Market update throughput: 11744680 upd/s
Is there support for pegged order types ? or support for limit orders that dynamically move with the best market quotes ?
Hi,
I'm building a simulator using your matching engine,
I was wondering if you have any suggestions on an efficient way to extract a time series (or just an event-by-event vector) of transacted (matched) price and quantity pairs ? In particular, when the matching is enabled at all time.
Best,
Is there commissioned work or paid support for this open source product?
Hello @chronoxor
I want to know why the price and quantity don't have a decimal value, is there any particular concern? is there any problem if I change it to decimal value?
Thank you in advance.
Hi @chronoxor,
While we wanted to add a market pair with 10 letters, we recently noticed that Symbol name is defined as 8 bytes.
What we thought could be an easy fix (just make char Name[16]
) seems to have implication as there are some places where code is optimized with unrolling and bit manipulations.
Before we patch it (and eventually propose it upstream) we wanted to clarify if that kind of change could impact code mechanical sympathy and as we are not high performance cpp expert by any means, wanted you opinion on the matters
Regards
I find a lot of CppCommen:: in the source code, but i cann't find the namespace in the project. is it a library? how can I get it? thanks
Matching engine benchmark results are missing from README?
===============================================================================
test cases: 18 | 16 passed | 2 failed
assertions: 151 | 149 passed | 2 failed
Hello,
I just cloned the project and followed the steps to build the project. I'm on macOS and run the command ./unix.sh
inside the build folder which produces an error: ./01-generate.sh: line 6: cmake: command not found
Any ideas on what could be going on?
Thanks
I am consistently getting a segfault after some millions of calls to AddOrder
:
Thread 6 "my_server" received signal SIGSEGV, Segmentation fault.
(gdb) backtrace
#0 0x00007fff803c9b2c in std::__1::pair<unsigned long, CppTrader::Matching::OrderNode*>::pair<unsigned long&, CppTrader::Matching::OrderNode*&, false>(unsigned long&, CppTrader::Matching::OrderNode*&) ()
#1 0x00007fff803bad6e in std::__1::pair<std::__1::__unwrap_ref_decay<unsigned long&>::type, std::__1::__$nwrap_ref_decay<CppTrader::Matching::OrderNode*&>::type> std::__1::make_pair<unsigned long&, CppTrader::M$tching::OrderNode*&>(unsigned long&, CppTrader::Matching::OrderNode*&) ()
#2 0x00007fff803b74c3 in CppTrader::Matching::MarketManager::AddLimitOrder(CppTrader::Matching::Order co$st&, bool) ()
#3 0x00007fff803b6bad in CppTrader::Matching::MarketManager::AddOrder(CppTrader::Matching::Order const&)
()
It looks like it is this call.
I have many threads that are receiving orders, creating CppTrader::Matching::Order
objects, and placing them into a FIFO concurrent queue: orders_queue.enqueue(order)
. However, reads from the queue and calls to AddOrder
are only ever done by a single thread:
Order order;
while (orders_queue.try_dequeue(order)) { // Empty the queue
if (std::this_thread::get_id() != this_thread_id) {
abort();
}
market.AddOrder(order);
}
Something strange I notice is that the segfault occurs at about the same point in execution on every run. I print some order book stats every 5 seconds, and it's always around the same number of orders processed before segfault:
// Run 1:
Matching Engine Stats:
Order Queue Depth: 0
Total Orders Count: 2027629
Total Executed Orders Count: 3170152
Orders Since Last Stats: 86721
OrderBook(Symbol=Symbol(Id=1; Name="MYSYMBOL"); Bids=237; Asks=240; BuyStop=0; SellStop=0; TrailingBuyStop=0; TrailingSellStop=0)
// Run 2:
Matching Engine Stats:
Order Queue Depth: 0
Total Orders Count: 2070242
Total Executed Orders Count: 3236818
Orders Since Last Stats: 88030
OrderBook(Symbol=Symbol(Id=1; Name="MYSYMBOL"); Bids=257; Asks=238; BuyStop=0; SellStop=0; TrailingBuyStop=0; TrailingSellStop=0)
// Run 3:
Matching Engine Stats:
Order Queue Depth: 0
Total Orders Count: 2048591
Total Executed Orders Count: 3202352
Orders Since Last Stats: 87628
OrderBook(Symbol=Symbol(Id=1; Name="MYSYMBOL"); Bids=254; Asks=239; BuyStop=0; SellStop=0; TrailingBuyStop=0; TrailingSellStop=0)
All orders are either buy or sell limit orders. The price and quantity values are randomly generated ints from 1 to 1000.
Do you have any recommendations on how I might be incorrectly handling the memory management of my Order
s, or other ideas on what might cause this?
Are newer C++11 to C++17 standard features used in CppTrader? What is the minimum version?
Dear @chronoxor
i'm using cpptrader-example-matching_engine from commit 43b8cfbc438373d3fb9b784025f6018f453e4e72
I have a Trailing Stop Limit Scenario as shown below :
add symbol 1 BTC_ETH
add book 1
enable matching
add limit sell 1 1 1000 100
add limit sell 2 1 1100 100
add limit sell 3 1 1100 100
add limit buy 4 1 900 100
add trailing stop-limit sell 5 1 700 700 100 100 0
add limit buy 6 1 1000 100
add limit buy 7 1 1100 100
add limit sell 8 1 900 100
add limit buy 9 1 1100 200
At first attempt the Trailing Stop is working and Executed. But When I retry with same scenario (without re-running the executable) the Trailing Stop Limit is not working. And why SellStop value is so big ?:
add limit sell 1 1 1000 100
add limit sell 2 1 1100 100
add limit sell 3 1 1100 100
add limit buy 4 1 900 100
add trailing stop-limit sell 5 1 700 700 100 100 0
add limit buy 6 1 1000 100
add limit buy 7 1 1100 100
add limit sell 8 1 900 100
add limit buy 9 1 1100 200
I will attach the screen shot also to make it easier to see / debug
Thank You very much
Have a nice day
Best Regards
Dear @chronoxor
I would like to ask :
Thank You very much for Your Great repo
Have a nice day
Best Regards
Rendy
Hi
I am trying to port CppTrader as library in Cython 3.0.
I have many problems to build this lib in Cython.
Is there any resources for build in Cython?
Maybe we need cancel the order function.
Hi chronoxor,
Thank you for the great order book implementation. Recently I have been playing with it to compare different approaches to order book creations with NYSE Arca data. Your approach was generic, so I have been tempted to adapt it.
At this moment I've been mostly interested in having L1 updates (specifically top of the book for each instrument). Implemented auxilliary structure to track if each update price changed worked just fine, e.g.:
`
OrderBook* orderBookPtr = (OrderBook*)market.GetOrderBook(instrument_id);
uint64_t orig_best_ask_price = 0;
uint64_t orig_best_ask_vol = 0;
if (orderBookPtr->best_ask())
{
orig_best_ask_price = orderBookPtr->best_ask()->Price;
orig_best_ask_vol = orderbookPtr->best_ask()->TotalVolume;
}
// Perform order book calculation e.g.
market.AddOrder(Order::Limit(1234567890, GetInstId("WMT"), OrderSide::SELL, 120000, 25);
if (orig_best_ask_price != orderBookPtr->best_ask()->Price || orig_best_ask_vol != orderBookPtr->best_ask()->TotalVolume)
{
// Update top of book
}
`
The code above works perfectly at least for the following updates: 'Modify', 'Execute', 'Delete', 'Add', and 'Replace'. However, it introduces latency, each check is required if the top of the book has been changed.
Investigating code by examples, I've been intrigued, if MarketHandler interface could be used to make code faster - https://github.com/chronoxor/CppTrader/blob/master/include/trader/matching/market_handler.h
However testing code, looks like 'UpdateLevel'
method 'LevelUpdate' structure 'Top' member is 'true' for non L1 data. Specifically looking at:Is this expected?
I have made some modifications for this code to work for L1 top of book updates and the moment still under testing (and at this moment isn't best effort implementation). But I was more wondering, if I missing a point here what is meant by 'LevelUpdate' member 'top', is it for multiple level depth? It's not clear though, how to know which LevelUpdate 'top' is relating to during callback... Or is it not expected & simply a bug?
To replicate a problem (currently only using NYSE data):
`
#ifndef NYSEFH_SRC_ORDERBOOK_MARKET_HANDLER_EXAMPLE
#define NYSEFH_SRC_ORDERBOOK_MARKET_HANDLER_EXAMPLE
#include <unordered_map>
#include <string>
#include "trader/matching/market_manager.h"
using namespace CppTrader::Matching;
class NYSEMarketHandler: public MarketHandler
{
public:
void onUpdateOrderBook(const OrderBook& orderBook, bool isTop) override
{
if (isTop)
{
static int num{ 0 };
printf("top of the book update! %i\n", num++);
}
}
};
NYSEMarketHandler marketHandler;
MarketManager market(marketHandler);
int main()
{
printf("Start cpptrader testing\n");
std::unordered_map<std::string, uint32_t> instKey {
{ "WMT", 1 }
};
Symbol sym{ instKey["WMT"], "WMT" };
market.AddSymbol(sym);
market.AddOrderBook(sym);
printf("in here#1\n");
market.AddOrder(Order::Limit(1234567890, instKey["WMT"], OrderSide::SELL, 23000, 15));
printf("in here#2\n");
market.AddOrder(Order::Limit(1234567891, instKey["WMT"], OrderSide::SELL, 25000, 25));
printf("in here#3\n");
market.ExecuteOrder(1234567891, 25);
printf("in here#4\n");
printf("Finish cpptrader testing\n");
return EXIT_SUCCESS;
}
#endif
`
Output:
Start cpptrader testing
in here#1
top of the book update! 0
in here#2
in here#3
top of the book update! 1
in here#4
Finish cpptrader testing
Expected output:
Start cpptrader testing
in here#1
top of the book update! 0
in here#2
in here#3
in here#4
Finish cpptrader testing
Thanks
gcc
complains when including the bfd.h
header:
In file included from /home/winterreise/Github/CppTrader/modules/CppCommon/source/system/stack_trace.cpp:21:
/usr/include/bfd.h:35:2: error: #error config.h must be included before this header
35 | #error config.h must be included before this header
| ^~~~~
make[2]: *** [modules/CppCommon/CMakeFiles/cppcommon.dir/build.make:433: modules/CppCommon/CMakeFiles/cppcommon.dir/source/system/stack_trace.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:663: modules/CppCommon/CMakeFiles/cppcommon.dir/all] Error 2
make: *** [Makefile:160: all] Error 2
Hi Ivan
Can someone leave better documentations in each classes, structs and methods. There are those of us who have trouble reading code and want to pursue the black box approach.
cd build
vs.bat
i can not find where vs.bat is. should you help me
/CppTrader/modules/CppBenchmark/modules/HdrHistogram/src/hdr_histogram_log.c:36:9:
error:
unknown pragma ignored [-Werror,-Wunknown-pragmas]
#pragma ide diagnostic ignored "readability-redundant-declaration"
Hello and thanks for putting together this great repository :). I noticed while reading https://github.com/chronoxor/CppTrader/blob/master/performance/market_manager_optimized_aggressive.cpp that many portions are substantially similar to an implementation I published, https://github.com/charles-cooper/itch-order-book/blob/master/order_book.h, but with no attribution. I recognize that you have licenses and copyrights on your own code, and I'm sure you would like others to respect those, so please do your part in upholding the tenets of the open source community and follow the terms of my license https://github.com/charles-cooper/itch-order-book/blob/master/LICENSE :).
Build command:
cmake.exe --build C:\Users\Asus\Documents\Projects\cpp\CppTrader\cmake-build-debug --target cpptrader-tests -- -j 9
Output error
modules/CppCommon/libcppcommon.a(path.cpp.obj): In function `CppCommon::Path::home()':
C:/Users/Asus/Documents/Projects/cpp/CppTrader/modules/CppCommon/source/filesystem/path.cpp:970: undefined reference to `__imp_GetUserProfileDirectoryW'
C:/Users/Asus/Documents/Projects/cpp/CppTrader/modules/CppCommon/source/filesystem/path.cpp:973: undefined reference to `__imp_GetUserProfileDirectoryW'
modules/CppCommon/libcppcommon.a(uuid.cpp.obj): In function `CppCommon::UUID::Sequential()':
C:/Users/Asus/Documents/Projects/cpp/CppTrader/modules/CppCommon/source/system/uuid.cpp:123: undefined reference to `__imp_UuidCreateSequential'
modules/CppCommon/libcppcommon.a(uuid.cpp.obj): In function `CppCommon::UUID::Random()':
C:/Users/Asus/Documents/Projects/cpp/CppTrader/modules/CppCommon/source/system/uuid.cpp:179: undefined reference to `__imp_UuidCreate'
collect2.exe: error: ld returned 1 exit status
mingw32-make.exe[3]: *** [CMakeFiles\cpptrader-tests.dir\build.make:158: cpptrader-tests.exe] Error 1
mingw32-make.exe[2]: *** [CMakeFiles\Makefile2:498: CMakeFiles/cpptrader-tests.dir/all] Error 2
mingw32-make.exe[1]: *** [CMakeFiles\Makefile2:505: CMakeFiles/cpptrader-tests.dir/rule] Error 2
Cmake output:
-- The C compiler identification is GNU 8.1.0
-- The CXX compiler identification is GNU 8.1.0
-- Check for working C compiler: C:/ProgramData/chocolatey/lib/mingw/tools/install/mingw64/bin/gcc.exe
-- Check for working C compiler: C:/ProgramData/chocolatey/lib/mingw/tools/install/mingw64/bin/gcc.exe - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/ProgramData/chocolatey/lib/mingw/tools/install/mingw64/bin/g++.exe
-- Check for working CXX compiler: C:/ProgramData/chocolatey/lib/mingw/tools/install/mingw64/bin/g++.exe - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- The System identification is Windows-10.0.19041 Windows 10.0.19041 Platform/Windows
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- The System identification is Windows-10.0.19041 Windows 10.0.19041 Platform/Windows
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - done
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- The System identification is Windows-10.0.19041 Windows 10.0.19041 Platform/Windows
-- Could NOT find DbgHelp (missing: DBGHELP_LIBRARY DBGHELP_INCLUDE_DIR)
-- Could NOT find RPC (missing: RPC_LIBRARY RPC_INCLUDE_DIR)
-- Could NOT find Userenv (missing: USERENV_LIBRARY USERENV_INCLUDE_DIR)
-- Found LIBVLD: C:/Users/Asus/Documents/Projects/cpp/CppTrader/modules/CppCommon/modules/vld/vld.lib
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/Asus/Documents/Projects/cpp/CppTrader/cmake-build-debug
Hi,
I just installed the code and was giving it a try but got the "killed" message when trying the optimized version of the market_manager (the non_optimized version runs without a problem).
Any idea of what is the problem?
Cheers
PS: all tests ran successfully.
Hi @chronoxor thank You for sharing this great matching engine library.
I would like to ask about AON (All Or None) Order. I'm using an example app "cpptrader-example-matching_engine".
Below is a sequence of order that i enter:
Now all orders executed, Order Id 2 has executed price = 99999999.
But for Order Id 3 and 4 the executed price are also 99999999.
For Order Id 3 and 4 it should be executed with price = 5 right ?
Below is a screen shot of the example app.
Hi I would like to ask, how to copy dir CppTrader to different location ? eg : I have done some modification of CppTrader in host PC & was run unix.sh at host PC. but now i want to copy into docker image & compile my code + library CppTrader inside docker build.
due to it has symbolic link, the link become invalid when copied to different path.
Currently my approach is copy modified CppTrader to /tmp -> then git init -> git clone from /tmp to src/libraries/CppTrader -> gil update -> unix.sh again. But this method require additional time to re clone all submodules.
Is there any best way to do it ?
Thank You very much
I am not that expert in c++, so looking for example to build limit order book from ITCH file for particular ticker. It will make jobs of other people much easier.
gcc
complains when including the bfd.h
header:
In file included from /home/winterreise/Github/CppTrader/modules/CppCommon/source/system/stack_trace.cpp:21:
/usr/include/bfd.h:35:2: error: #error config.h must be included before this header
35 | #error config.h must be included before this header
| ^~~~~
make[2]: *** [modules/CppCommon/CMakeFiles/cppcommon.dir/build.make:433: modules/CppCommon/CMakeFiles/cppcommon.dir/source/system/stack_trace.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:663: modules/CppCommon/CMakeFiles/cppcommon.dir/all] Error 2
make: *** [Makefile:160: all] Error 2
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.