ryanphuang / imembench Goto Github PK
View Code? Open in Web Editor NEWGeneric benchmarks for various in-memory storage systems
Generic benchmarks for various in-memory storage systems
When I try to run the following command in the kvtracer directory
mvn -pl kvtracer -am clean package
I get the output shown here: http://pastebin.com/DuvV5vLe
My directory structure is:
ycsb-0.7.0/kvtracer
Encounter runtime failures after running the instrumented master node (using java-allocation-instrumenter-3.0).
Exception in thread "main" java.lang.IllegalArgumentException: Strange Collection type java.util.ArrayList: can not determine type parameters
at com.fasterxml.jackson.databind.type.TypeFactory._collectionType(TypeFactory.java:904)
at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:722)
at com.fasterxml.jackson.databind.type.TypeFactory._constructType(TypeFactory.java:371)
at com.fasterxml.jackson.databind.type.TypeFactory.constructType(TypeFactory.java:342)
at com.fasterxml.jackson.databind.cfg.MapperConfig.constructType(MapperConfig.java:241)
at com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:442)
at com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:599)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:93)
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2682)
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2630)
at tachyon.master.JsonObject.withParameter(JsonObject.java:110)
at tachyon.master.ImageElement.withParameter(ImageElement.java:53)
at tachyon.master.InodeFolder.writeImage(InodeFolder.java:256)
at tachyon.master.MasterInfo.writeImage(MasterInfo.java:2417)
at tachyon.master.Image.create(Image.java:59)
at tachyon.master.Journal.createImage(Journal.java:78)
at tachyon.master.MasterInfo.init(MasterInfo.java:1780)
at tachyon.master.TachyonMaster.setup(TachyonMaster.java:216)
at tachyon.master.TachyonMaster.start(TachyonMaster.java:277)
at tachyon.master.TachyonMaster.main(TachyonMaster.java:73)
I wanna invoke a discussion on how to measure the allocation overhead of Tachyon.
Foremost, according to my understanding, only the allocation overhead of workers (who stores real data) matters; the master only maintains metadata so it's allocation overhead is not interesting. Correct me if I'm wrong @stormspirit, @jakemask
I managed to apply the java-allocation-instrumenter (https://github.com/google/allocation-instrumenter) on Tachyon workers, and passed all the tests.
However, I encountered the difficulties to analyze the recorded overhead. First, the instrumenter does not tell the time overhead but only records the size of each allocated objects.
Second, the method used by instrumenter is to insert instructions after each new
statement, which does not tell the real allocation overhead. This is a general problem of instrumentation based on new
or malloc
Long story short, there are two big problems (I don't know how to solve),
Consider the following code snippets,
new String("foo");
new String("fooo");
new String("foooo");
new String("fooooo");
new String("foooooo");
new String("fooooooo");
And the logs show,
Allocated object foo of type java/lang/String with size 24
Allocated object fooo of type java/lang/String with size 24
Allocated object foooo of type java/lang/String with size 24
Allocated object fooooo of type java/lang/String with size 24
Allocated object foooooo of type java/lang/String with size 24
Allocated object fooooooo of type java/lang/String with size 24
Allocated object java.lang.Object@76724124 of type java/lang/Object with size 16
Allocated object [C@78b5a19a of type char with size 48 (an array of size 13)
Allocated object com.google.monitoring.runtime.instrumentation.asm.ClassReader@3988aba1 of type com/google/monitoring/runtime/instrumentation/asm/ClassReader with size 32
Allocated object [C@5d97b6d5 of type char with size 80 (an array of size 32)
Allocated object [C@8fb758c of type char with size 32 (an array of size 6)
Allocated object [C@250e364d of type char with size 80 (an array of size 30)
Allocated object [C@59a754ee of type char with size 32 (an array of size 6)
Allocated object [C@39667011 of type char with size 72 (an array of size 27)
Allocated object [C@6a6235a4 of type char with size 64 (an array of size 21)
Allocated object [C@3e0adafb of type char with size 24 (an array of size 3)
Allocated object [C@5de9e9f0 of type char with size 96 (an array of size 37)
Allocated object [C@5705fca4 of type char with size 136 (an array of size 59)
Allocated object [C@2cb12cf5 of type char with size 72 (an array of size 25)
Allocated object [C@4484f69a of type char with size 24 (an array of size 4)
Allocated object [C@56302193 of type char with size 24 (an array of size 1)
Allocated object [C@2ea8072d of type char with size 40 (an array of size 11)
Allocated object [C@bdb4089 of type char with size 64 (an array of size 21)
Allocated object [C@6808574e of type char with size 32 (an array of size 6)
Allocated object [C@3f63cf65 of type char with size 96 (an array of size 38)
Allocated object [C@511c26cb of type char with size 56 (an array of size 17)
......
From the first 4 lines, we can see the String
object actually have the same size. What differs is the field objects inside the strings. However, as the instrumenter does not provide any more informative interface, this is pretty much the only thing we can get. I don't know how to leverage such information, except do a type-based analysis.
new
statement.In fact, if Tachyon or any other system performs some pre-allocation (e.g., allocating a pool of memory dur ing initialization), and then return the memory blocks when the client requests files/blocks. The instrumenter would tell us nothing because there is no new
statements invoked at that time.
Luckly, in Tachyon, memory allocation related code is well maintained in the directory
core/src/main/java/tachyon/worker/allocation
They implemented three allocation strategy: RR, MaxFree, and Random, all of them inherits the abstract class AllocateStrategyBase
.
Basically, the only method requires to implement is getStorageDir()
which
33 /**
34 * Allocate space on StorageDirs. It returns the affordable StorageDir according to
35 * AllocateStrategy, null if no enough space available on StorageDirs.
36 *
37 * @param storageDirs candidates of StorageDirs that space will be allocated in
38 * @param userId id of user
39 * @param requestSizeBytes size to request in bytes
40 * @return StorageDir assigned, null if no StorageDir is assigned
41 */
Though the algorithm of selecting the affordable StorageDir are different, the key operation is ``requestSpace()` which get the space from the affordable storage.
So my idea is to add time counters in getStorageDir()
so that we can know the allocation overhead.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.