Comments (5)
@cryptix i will look into this later with more detail and propose a solution or fix the bug if so.
however, judging from what you need, i would suggest a custom binary (de)serialization for this unless you necessarily need it to be json for readability? binary would allow make the (de)serialization a lot faster and more space efficient, etc. you could also implement custom compression this way to make it even smaller (varints, delta-compression, etc.).
i intended to implement a binary (de)serialization for all the structures, but unfortunately never found the time.
i frankly have no strong opinion on how to json-encode binary data and i am not sure if there a "one shoe fits all" solution to this. another issue i am having is if i encode it as string (e.g. exact hexadecimal representation or base64, etc.) then how would the deserialization know that this should be deserialized as an array of bytes, it only sees it as a string?
this all could be solved with the binary (de)serialization i was planning on doing that would hold some metadata internally in order to deserialize to the same types that were serialized.
from gods.
most serialization is done using the native (de)serialization, e.g. https://github.com/emirpasic/gods/blob/master/lists/arraylist/serialization.go , however in some cases i needed more control of the (de)serialization and needed to custom implement it. this implementation does not differ much from how other libraries implement it, it's really nothing magical about it, just basic string/binary manipulations.
could you give me a simple example, so i can reproduce your problem and please describe what the expected output should be.
i am not sure of what you mean by 32-byte hashes, but i'll assume you are talking about an array of bytes? there isn't a general consensus on how to (de)serialize binary data into JSON. you mention base64, why? it could also be something else, e.g. base32 or hexadecimal string representation. this project tries to stay unopinionated with that regard, flexible and generic, i.e. if you need base64 (i like it too), but that doesn't mean that everyone is working with base64.
there are examples here how to implement a custom (de)serialization that suites your needs and i can point you in the right direction if you'd like and if i understood your problem.
cheers
from gods.
Hey @emirpasic thanks for the quick response! I only mentioned base64 since that is what the stdlib does when it encounters a []byte
in a struct field when marshaling to JSON. I wouldn't mind a more compact format, gob would also suffice for instance. I just need it to persist for my own application when shutting down/starting again, it's not meant as an exchange format or anything.
Here is some contrived code with hardcoded examples of what I tried before opening the issue. I just shortened the values because that doesn't mattter.
func main() {
// returns a []byte of the same size as the keys for the tree
msg, err := ssb.ParseMessageRef(os.Args[1])
check(err)
tree := btree.NewWith(3, bytesCompare)
b, err := ioutil.ReadFile("tree.json")
if os.IsNotExist(err) {
fmt.Println("rebuilding tree...")
ctx := context.TODO()
vals := []struct {
Hash []byte
Int int
}{
{[]byte{1, 2, 3, 4, 5}, 23},
{[]byte{6, 6, 6, 6, 6}, 42},
{[]byte{0, 0, 0, 0, 1}, 131},
{[]byte{0, 0, 0, 0, 2}, 555},
}
start := time.Now()
i := 0
for i, val := range vals {
tree.Put(val.Hash, val.Int)
i++
}
fmt.Printf("building took %v for %d elements\n", time.Since(start), i)
fmt.Println("size:", tree.Size())
b, err := tree.ToJSON()
check(err)
ioutil.WriteFile("tree.json", b, 0700)
} else {
err = tree.FromJSON(b)
check(err)
fmt.Println("loaded existing tree")
}
kv, ok := tree.Get(msg.Hash)
fmt.Println(ok, kv)
}
func bytesCompare(a, b interface{}) int {
bytesA := a.([]byte)
bytesB := b.([]byte)
return bytes.Compare(bytesA, bytesB)
}
from gods.
Yup, binary would totally work for me! If you point me to an example of that I'm sure I can take the next steps by myself.
from gods.
here is one of the few examples where i had to implement a custom serialization in order to have more control over ordering of keys in an ordered map and the default behavior did not suffice.
https://github.com/emirpasic/gods/blob/master/maps/linkedhashmap/serialization.go
it basically writes and reads text (json), so it should be easy to implement reading/writing of binary data instead. as you suggested, make use of the native BinaryMarshaler and BinaryUnmarshaler to conform to the go's ideology and keep things simple.
please let me know how this goes and paste the code here so we can make use of it.
from gods.
Related Issues (20)
- circularbuffer values is not concurrent safe HOT 1
- priorityqueue question HOT 1
- What scenarios will this project be imported HOT 4
- Coverity Scan - Explicit null dereferenced HOT 1
- hashbidimaps Values() returns keys HOT 2
- Why not replace "%+v" to json.Marshal in utils.ToString
- Does it make sense to add goroutine / thread safe data structures for all the existing data structures? HOT 1
- arraylist.List should not maintain size HOT 1
- Add a Ordered linked HOT 2
- Hey! I am interested in translating this project into Portuguese, Brazil. Can I contribute? HOT 1
- I would like to know how to delete a node data while treemap loop read. I want to know where is my code wrong? HOT 2
- Add to Go official wiki? HOT 3
- generic upgrade request HOT 3
- RedBlackTree: Iterators become invalid after removing an element. HOT 2
- DS which can give element if present or next greater if not present, elements should be stored in sorted order HOT 2
- LinkedHashMap Sort? HOT 3
- Multiset support
- deque support
- hashset should support NewWith Comparator
- Add method of Set lack info about the insert take place or not
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gods.