Comments (13)
Thanks for bringing this up. Use the previous version 2.6.1 until I can figure out the issue. I just updated the string writing algorithm in 2.6.2 which probably introduced the issue.
from glaze.
I added a test for your data here: #998, but it passes so there's something deeper going on. I don't think this is a memory corruption issue, but probably a missing detection of a quote character. I'm trying to dig deeper.
If you're able to let me know what types you're using in this example that might help.
Here is the test code I added. Let me know if I should change anything:
struct price_t
{
uint64_t price{};
uint64_t volume{};
};
struct ticker_t
{
uint64_t time{};
std::string exchange{};
std::string symbol{};
std::vector<price_t> asks{};
std::vector<price_t> bids{};
uint64_t price{};
uint64_t volume{};
uint64_t open_interest{};
uint64_t ceiling{};
uint64_t floor{};
};
suite ticker_tests = [] {
"ticker"_test = [] {
std::string_view json = R"({
"time": 1686621452000000000,
"exchange": "SHFE",
"symbol": "rb2310",
"asks": [
{
"price": 3698,
"volume": 2882
}
],
"bids": [
{
"price": 3693,
"volume": 789
}
],
"price": 3693,
"volume": 820389,
"open_interest": 1881506,
"ceiling": 4075,
"floor": 3268
})";
ticker_t obj{};
const auto ec = glz::read_json(obj, json);
expect(!ec) << glz::format_error(ec, json);
std::string s = glz::write_json(obj);
expect(s == glz::minify_json(json)) << s;
};
};
from glaze.
I can't seem to reproduce your issue. If you could try to reproduce the bug at your leisure that would be very helpful, but I'll continue to add more tests to try to locate an issue.
from glaze.
Hi Stephen
Thanks for your prompt response, I updated the glaze weeks ago, I will find out the update time of glaze main tomorrow , I hope the main is always the latest stable production branch, I have millions of records per day, I will try my best to locate the issue by adding some instrumental codes.
from glaze.
Thanks. main
should always be the latest stable state, as nothing gets merged with main
without passing unit tests. Glaze has thousands of tests, but my plan is to add more large datasets to the testing code to catch corner cases.
Feel free to submit merge requests for more test projects that mimic your use case, to help prevent issues arising.
from glaze.
I do remember merging recently into main and there being a bug like the one you saw. It may be that the current main and latest release are good.
In any case, I would recommend targeting specific releases, because this makes it easier to debug issues and the releases allow replication.
from glaze.
@sweihub, have you encountered any more issues or determined if this issue no longer exists?
from glaze.
Sorry I am engaged in another stuff these days, this issue should still exist, I’ll add some instrument codes and try to capture the issue. Because it rarely happens, so my ticker program continues to run.
Next time, I’ll lock down to a specific tag version, and not using the main in order to identify issues if any.
from glaze.
I added the instrmental code to verify the generated JSON, will deploy after trading hours, will get back to you if anything captured.
void verify(FILE *bf, core::rpc::Depth const &depth, std::string const &buf)
{
core::rpc::Depth output{};
if (json::read(output, buf) == 0)
return;
// BUG
fprintf(bf, "//--Data--------------------------------------\n");
fprintf(bf, "core::rpc::Depth depth = {\n");
fprintf(bf, ".time = %lldLL,\n", (long long)depth.time);
fprintf(bf, ".symbol = \"%s\",\n", depth.symbol.c_str());
fprintf(bf, ".exchange = \"%s\",\n", depth.exchange.c_str());
if (depth.price.has_value())
fprintf(bf, ".price = %f,\n", depth.price.value());
if (depth.volume.has_value())
fprintf(bf, ".volume = %f,\n", depth.volume.value());
if (depth.turnover.has_value())
fprintf(bf, ".turnover = %f,\n", depth.turnover.value());
if (depth.size.has_value())
fprintf(bf, ".size = %f,\n", depth.size.value());
if (depth.open_interest.has_value())
fprintf(bf, ".open_interest = %f,\n", depth.open_interest.value());
if (depth.ceiling.has_value())
fprintf(bf, ".ceiling = %f,\n", depth.ceiling.value());
if (depth.floor.has_value())
fprintf(bf, ".floor = %f,\n", depth.floor.value());
if (depth.iopv.has_value())
fprintf(bf, ".iopv = %f,\n", depth.iopv.value());
if (depth.open.has_value())
fprintf(bf, ".open = %f,\n", depth.open.value());
if (depth.high.has_value())
fprintf(bf, ".high = %f,\n", depth.high.value());
if (depth.low.has_value())
fprintf(bf, ".low = %f,\n", depth.low.value());
if (depth.close.has_value())
fprintf(bf, ".close = %f,\n", depth.close.value());
if (depth.prev.has_value())
fprintf(bf, ".prev = %f,\n", depth.prev.value());
// asks [{price, volume}, ...]
fprintf(bf, ".asks = {\n");
for (auto &ask : depth.asks) {
fprintf(bf, "{%f, %f},\n", ask.price, ask.volume);
}
fprintf(bf, "},\n");
// bids
fprintf(bf, ".bids = {\n");
for (auto &bid : depth.bids) {
fprintf(bf, "{%f, %f},\n", bid.price, bid.volume);
}
fprintf(bf, "},\n");
// end of data
fprintf(bf, "};\n\n");
fprintf(bf, "/* JSON\n");
fprintf(bf, "%s\n", buf.c_str());
fprintf(bf, "*/\n");
}
from glaze.
Awesome, thank you!
from glaze.
@sweihub Do you think this issue still exists?
from glaze.
Hi Dear Stephen,
I did not reproduce the issue although I added the instrumental code, let's close this issue for now! I will reopen if it occurs, hopefully not!
Thanks for your efforts!
from glaze.
Thanks, definitely submit a new issue if anything pops up. I'm just trying to stay on top of potential issue.
from glaze.
Related Issues (20)
- calculate serialize size before serialization and serialize to pre-allocated memory HOT 4
- Option to use partial with char* or char[] HOT 7
- data_must_be_null_terminated issue HOT 4
- Raw char buffer handling HOT 2
- Built-in support for type control HOT 3
- reflection fails with custom serializer for std::chrono types on clang/gcc HOT 7
- How do I write an enum as a raw string? HOT 1
- compile error with error_on_missing_keys HOT 1
- old GCC 11.4.0 compiler support HOT 8
- Link to NDJSON website looks sus... HOT 2
- [ICE] `msvc 17.8` (`glaze 2.8.1`) HOT 1
- Numeric variant deduction type
- Write_binary/read_binary support for the "json_t" interface HOT 6
- is_[json_type] concepts in glz:: namespace
- Compilation error when structure have more than 32 fields HOT 1
- How to pass glz::read_json to a std::function? HOT 1
- Support std::variant<...,std::variant<...> > HOT 2
- repe::registry needs to handle throws on function calls
- Tuple routines not working as expected HOT 1
- Does glaze handle structs with static members ? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from glaze.