Giter VIP home page Giter VIP logo

Comments (13)

stephenberry avatar stephenberry commented on June 16, 2024

Thanks for bringing this up. Use the previous version 2.6.1 until I can figure out the issue. I just updated the string writing algorithm in 2.6.2 which probably introduced the issue.

from glaze.

stephenberry avatar stephenberry commented on June 16, 2024

I added a test for your data here: #998, but it passes so there's something deeper going on. I don't think this is a memory corruption issue, but probably a missing detection of a quote character. I'm trying to dig deeper.

If you're able to let me know what types you're using in this example that might help.

Here is the test code I added. Let me know if I should change anything:

struct price_t
{
   uint64_t price{};
   uint64_t volume{};
};

struct ticker_t
{
   uint64_t time{};
   std::string exchange{};
   std::string symbol{};
   std::vector<price_t> asks{};
   std::vector<price_t> bids{};
   uint64_t price{};
   uint64_t volume{};
   uint64_t open_interest{};
   uint64_t ceiling{};
   uint64_t floor{};
};

suite ticker_tests = [] {
   "ticker"_test = [] {
      std::string_view json = R"({
     "time": 1686621452000000000,
     "exchange": "SHFE",
     "symbol": "rb2310",
     "asks": [
       {
         "price": 3698,
         "volume": 2882
       }
     ],
     "bids": [
       {
         "price": 3693,
         "volume": 789
       }
     ],
     "price": 3693,
     "volume": 820389,
     "open_interest": 1881506,
     "ceiling": 4075,
     "floor": 3268
   })";
      
      ticker_t obj{};
      
      const auto ec = glz::read_json(obj, json);
      expect(!ec) << glz::format_error(ec, json);
      std::string s = glz::write_json(obj);
      expect(s == glz::minify_json(json)) << s;
   };
};

from glaze.

stephenberry avatar stephenberry commented on June 16, 2024

I can't seem to reproduce your issue. If you could try to reproduce the bug at your leisure that would be very helpful, but I'll continue to add more tests to try to locate an issue.

from glaze.

sweihub avatar sweihub commented on June 16, 2024

Hi Stephen

Thanks for your prompt response, I updated the glaze weeks ago, I will find out the update time of glaze main tomorrow , I hope the main is always the latest stable production branch, I have millions of records per day, I will try my best to locate the issue by adding some instrumental codes.

from glaze.

stephenberry avatar stephenberry commented on June 16, 2024

Thanks. main should always be the latest stable state, as nothing gets merged with main without passing unit tests. Glaze has thousands of tests, but my plan is to add more large datasets to the testing code to catch corner cases.

Feel free to submit merge requests for more test projects that mimic your use case, to help prevent issues arising.

from glaze.

stephenberry avatar stephenberry commented on June 16, 2024

I do remember merging recently into main and there being a bug like the one you saw. It may be that the current main and latest release are good.

In any case, I would recommend targeting specific releases, because this makes it easier to debug issues and the releases allow replication.

from glaze.

stephenberry avatar stephenberry commented on June 16, 2024

@sweihub, have you encountered any more issues or determined if this issue no longer exists?

from glaze.

sweihub avatar sweihub commented on June 16, 2024

Sorry I am engaged in another stuff these days, this issue should still exist, I’ll add some instrument codes and try to capture the issue. Because it rarely happens, so my ticker program continues to run.

Next time, I’ll lock down to a specific tag version, and not using the main in order to identify issues if any.

from glaze.

sweihub avatar sweihub commented on June 16, 2024

I added the instrmental code to verify the generated JSON, will deploy after trading hours, will get back to you if anything captured.

void verify(FILE *bf, core::rpc::Depth const &depth, std::string const &buf)
{
    core::rpc::Depth output{};

    if (json::read(output, buf) == 0)
        return;

    // BUG
    fprintf(bf, "//--Data--------------------------------------\n");
    fprintf(bf, "core::rpc::Depth depth = {\n");
    fprintf(bf, ".time = %lldLL,\n", (long long)depth.time);
    fprintf(bf, ".symbol = \"%s\",\n", depth.symbol.c_str());
    fprintf(bf, ".exchange = \"%s\",\n", depth.exchange.c_str());

    if (depth.price.has_value())
        fprintf(bf, ".price = %f,\n", depth.price.value());
    if (depth.volume.has_value())
        fprintf(bf, ".volume = %f,\n", depth.volume.value());
    if (depth.turnover.has_value())
        fprintf(bf, ".turnover = %f,\n", depth.turnover.value());
    if (depth.size.has_value())
        fprintf(bf, ".size = %f,\n", depth.size.value());
    if (depth.open_interest.has_value())
        fprintf(bf, ".open_interest = %f,\n", depth.open_interest.value());
    if (depth.ceiling.has_value())
        fprintf(bf, ".ceiling = %f,\n", depth.ceiling.value());
    if (depth.floor.has_value())
        fprintf(bf, ".floor = %f,\n", depth.floor.value());
    if (depth.iopv.has_value())
        fprintf(bf, ".iopv = %f,\n", depth.iopv.value());
    if (depth.open.has_value())
        fprintf(bf, ".open = %f,\n", depth.open.value());
    if (depth.high.has_value())
        fprintf(bf, ".high = %f,\n", depth.high.value());
    if (depth.low.has_value())
        fprintf(bf, ".low = %f,\n", depth.low.value());
    if (depth.close.has_value())
        fprintf(bf, ".close = %f,\n", depth.close.value());
    if (depth.prev.has_value())
        fprintf(bf, ".prev = %f,\n", depth.prev.value());

    // asks [{price, volume}, ...]
    fprintf(bf, ".asks = {\n");
    for (auto &ask : depth.asks) {
        fprintf(bf, "{%f, %f},\n", ask.price, ask.volume);
    }
    fprintf(bf, "},\n");

    // bids
    fprintf(bf, ".bids = {\n");
    for (auto &bid : depth.bids) {
        fprintf(bf, "{%f, %f},\n", bid.price, bid.volume);
    }
    fprintf(bf, "},\n");

    // end of data
    fprintf(bf, "};\n\n");

    fprintf(bf, "/* JSON\n");
    fprintf(bf, "%s\n", buf.c_str());
    fprintf(bf, "*/\n");
}

from glaze.

stephenberry avatar stephenberry commented on June 16, 2024

Awesome, thank you!

from glaze.

stephenberry avatar stephenberry commented on June 16, 2024

@sweihub Do you think this issue still exists?

from glaze.

sweihub avatar sweihub commented on June 16, 2024

Hi Dear Stephen,

I did not reproduce the issue although I added the instrumental code, let's close this issue for now! I will reopen if it occurs, hopefully not!

Thanks for your efforts!

from glaze.

stephenberry avatar stephenberry commented on June 16, 2024

Thanks, definitely submit a new issue if anything pops up. I'm just trying to stay on top of potential issue.

from glaze.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.