Comments (6)
+= does an append, which is similar in performance to appending to a std::vector, and the fastest way to do string concatenation as far as I know (certainly better than ostringstream).
Why would it be copying the entire string? It typically only reallocates every power of two in size.
from flatbuffers.
Profiling a similar case showed a lot of overhead, but maybe it is 'just' the doubling of the buffer each time, instead of reallocing each time something is appended. Or I was confused about the exact meaning of +=
.
But there is still an optimization opportunity here: is there any way to estimate a reasonable amount of bytes beforehand to minimize the number of realloc needed?
Also I would like to see the option to direct output it to std::cout
.
from flatbuffers.
FlatBuffers is a very performance-conscious libary. That said, I don't think these reallocs will cause performance problems, even when generating large amounts of JSON. You could probably estimate an ideal buffer size given the size of the input FlatBuffer you're generating from.. this is something I could add, but I don't think it will make much of a difference.
If you want to output to cout, first generating the whole string would still be faster than calling cout for every substring.
from flatbuffers.
I agree. I was confused and too quick to judge. My apologies.
I still would like to see a streaming API to deal with large datasets on memory constrained systems (both reading and writing), maybe I will open a different ticket for it (as it not performance related).
from flatbuffers.
You're right, it would be useful for that. Though I would hope that if you program for a memory constrained system, you don't use the text representation at all, and work with binary only.
from flatbuffers.
I have one project that depends on an external JSON REST API, which sends over 1.5 Mb of decompressed JSON. The devices that should be supported are Android phones with 20 Mb of heap space available (so a continuous block of 1.5 Mb is almost impossible). Only because of the streaming APIs it is possible to even run on such devices. I can also imagine the use on constrained embedded systems like Arduino.
from flatbuffers.
Related Issues (20)
- Streaming flatbuffers in java HOT 1
- [TS] optional escalar not set in the binary when it holds value 0 HOT 13
- Release a new version, please HOT 1
- Default value of 0 for field is not part of [optional] enum (java)
- Buffer generated in python gets rejected by flatcc [Python / C, Flatbuffer 23.5.26]
- Invalid Code generation if Property has same name as Struct [Windows/C#/24.3.7]
- v24.3.7 of Rust crate not released to crates.io HOT 4
- [v24.3.7] Rust: push_u8_u64_alignment test fails on i686 target (Ubuntu 22.04, CentOS 9)
- How to print nested table using FlatBufferToString?
- [Python] Flatc stub compatibility with gRPC asyncio Interface
- [Rust] Add better support for "crate-per-schema" HOT 2
- [Bazel CI] Swift toolchain analysis failed due to missing 'swift' configuration fragment HOT 1
- [C#][Feature Request] Unity Game Development needs mutate fixed size string value HOT 2
- Generated CreateXDirect with std::span<T> or "T* + size" payloads
- newbie's question.. use in c# / vector struct / adding int64...
- Feature: --gen-object-api support for Kotlin [Kotlin, FlatBuffers 24.3.25]
- [Rust] inconsistent nested_flatbuffers validations
- [.NET] Target .NET Standard 2.0 again
- Missing type annotations in Python Library
- [TS] flatc silently crashing (segmentation fault) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flatbuffers.