chadaustin / buffer-builder Goto Github PK

Plan:

Have an internal 'dead' bit in BufferWriter

bw_new return 0 on failure, checked on Haskell side
bw_append functions don't fail, they just mark as dead on allocation failure
bw_get_and_trim returns failure code, Haskell throws memory error if dead

The newly added Semigroup instances are currently not compatible with GHC <= 7.10.

https://matrix.hackage.haskell.org/package/buffer-builder

Would it be possible to fix the lower bounds for buffer-builder-0.2.4.5 on Hackage so that dependency solvers can catch it?

On naive implementations of realloc(), the current growth implementation copies (lg N) times. If we chunked into a linked list, we'd have the option to produce a Lazy ByteString, avoiding a copy, or we could copy once into a Strict ByteString.

I can't imagine a situation where this matters yet, but in case it comes up someday:

lpsmith/json-builder@b33adc7

Also http://blog.melding-monads.com/2012/02/24/implementing-json-builder-or-how-the-commabuilder-got-its-spots/

This fails to build in the absence of libstdc++. Looking at the source code it doesn't look like the c-bits actually use any C++ functionality.

Is it trivial to adapt deriveJson from Aeson to automatically derive ToJson instances? I looked into using buffer builder for a project but got stopped in tracks because I use automatic derivation everywhere (instead of hand-coding it tediously). An example derivation is like this through Data.Aeson.TH:
deriveJSON defaultOptions{fieldLabelModifier = Prelude.drop 5,constructorTagModifier = Prelude.map toLower,omitNothingFields = True} ''ARecordType

It seems that in order to use encodeJson, ToJson instance must be provided which must be hand-coded for now. If it can somehow be derived from Data.Aeson.ToJSON, then it will make it very useful!

Unless I missed something about how to adapt an existing Aeson ToJSON to ToJson?

It appears the header file branchlut.h is missing: http://hydra.cryp.to/build/577987/nixlog/2/raw?

I've got some code where the buffered value needs to exceed 1MB before being flushed over the network, and I'm currently having to do my own bookkeeping of the size. I'm pretty sure that the builder tracks its utilization under the hood, so would it be realistic to make that data publicly accessible?

Sorry that title isn't very clear. My use case is described here. Essentially I need to be able to do something like:

writeBuilderOnto :: Builder -> ByteArrayThingIMightHaveToGrow -> IO ()

bytestring's Builder almost gets me there with the internal BufferWriter, but unfortunately we can determine the number of bytes that will be written ahead of time from just the Builder. This is a hard requirement for me.

buffer-builder looks like an attractive place for this functionality to live because I think nearly all the instances you provide have O(1) length already, and it looks like your library is super fast to boot. I think only lazy bytestrings and the String case would need to force the spine to get an O(n) length, which needs to happen anyway.

Does that analysis seem right, and make sense? And would you be open to:

1. Limiting BufferBuilder such that its allocation area is known before running
2. Providing a lower-level BufferWriter-like function that I can run on a pre-allocated buffer

Let me know what you think, and I may hack on this in the next couple months if it's the sort of thing you'd be interested in having here.

The haskell-perf team has a list of benchmarks of different bytestring builders. buffer-builder should get added there:
https://github.com/haskell-perf/strict-bytestring-builders