retrieva / pficommon Goto Github PK

std::swap in mmapper.h#L84 can be multiple definition error when include <algorithm>.

https://github.com/pfi/pficommon/blob/master/src/visualization/ppm_test.cpp#L132

Currently, pfi::lang::lexical_cast don't throw any exception for invalid input which has "valid prefix". (e.g. lexical_cast<int>("0.5") )
But caller have no way to check whole of input is valid? or only prefix is valid?
So lexical_cast should throw exception for such input.

And also we need some tests for cast.h to make the spec clear.

In json_object, I sometimes want to merge two json_object.

json json1 // {"hoge": {...}}
json json2 // {"foo": 1, "bar", true}
json1.merge(json2) //  {"foo": 1, "bar", true, "hoge": {...}}

What do you think?

src/lang/wscript forgets to describe installing demangle.h

I have following code:

#include <pficommon/text/json.h>
#include <string>
#include <vector>

using namespace std;
using namespace pfi::text::json;

int main()
{
  json hoge(new json_object());
  vector<string> strs(json_cast<vector<string> >(hoge));

  return 0;
}

Current origin/master works.
But old version(e.g. pficommon 1.3.1) raises json_bad_cast.

$ ./a.out
terminate called after throwing an instance of 'pfi::text::json::json_bad_cast<std::vector<std::string, std::allocator<std::string> > >'
  what():  Failed json_cast from N3pfi4text4json11json_objectE to St6vectorISsSaISsEE.
zsh: abort      ./a.out

Older behavior seems to be correct.
"{}" should not be converted to vector, it's a map.

P.S.

d0d7361

This commit introduces new behavior.

Current serialization and endian_util does not support floating point and boolean. Their formats depend on machine or compiler environment.

Release new version, or update version number of repository for other project like jubatus/jubatus.

• Now master/HEAD(30dd4f8) includes many bug fixes and improvements from v1.3.1.0.
• There is no milestone / release schedule

stringized result of json_float does not have '.' if json_float has an integer value.

std::vector<double> vec;
vec.push_back(1.1);
vec.push_back(0);
vec.push_back(2.5);

std::ostringstream oss;
oss << pfi::text::json::to_json(vec);
std::cout << oss.str() << std::endl;  // [1.1, 0, 2.5];

This format is not useful, bacause following code raises an exception.

pfi::text::json::json j;
std::istringstream iss("[1.1,0,2.5]");
iss >> j;
std::vector<double> arr;
try {
  pfi::text::json::from_json(j, arr);
} catch (const std::exception& e) {
  std::cout << e.what() << std::endl;
}

json_float should generate a string with decimal point.

P.S.

In Ruby, to_json generates a string with decimal point.

{'hoge' => 1.0}.to_json  #=> "{\"hoge\":1.0}"

pficommon does not build on OS X 10.9 or later, because it uses the transitional "tr1" namespace which is no longer available in OS X 10.9 or later:

pficommon-26b6655092adc95649c39706b37502e466e5efe9/src/concurrent/../lang/function.h:35:10: fatal error: 'tr1/functional' file not found
#include <tr1/functional>
         ^
1 error generated.

Note: if you want to retain compatibility with OS X 10.8 or earlier, you have to continue using "tr1" on those systems. So you'll have to detect whether "tr1" exists or not and act accordingly.

"[-]" is parsed to [0].

pficommon currently uses branch based versioning.
But, adding tag is helpful for some package manages like dep and rpm.

So, we should add a versioning tag at each release.

Now, we can convert between pfi::text::json::json and basic types. (int, double, bool, string, map<string, T>, vector)

But, we cannot convert between pfi::text::json::json and float.
For example, following code result in compilation error.

struct SomeUserDefinedType {
  float f;

private:
  pfi::data::serialization::access;
  template<typename Ar>
  void serialize(Ar& ar) {
    ar & MEMBER(f);
  }
};

SomeUserDefinedType value;
pfi::text::json::json js = pfi::text::json::to_json(value);

So, I would like to support convertion between pfi::text::json::json and float type.

The convertion from pfi::text::json::json to float is:

pfi::text::json::json js(new pfi::text::json::json_float(3.14));

float f;
pfi::text::json::from_json<float>(js, f);

The convertion from float to pfi::text::json::json is:

float f = 3.14;
pfi::text::json::json js = pfi::text::json::to_json(f)

Should we support such convertion? And, can we implement such convertion?

Inclusion of <unistd.h> is deleted in gcc 4.7 (http://gcc.gnu.org/gcc-4.7/changes.html).
Due to this, gcc could't find declaration of ssize_t in system/file.h

[Environment]

• python 2.6.6
• gcc 4.7.1

[Compiler Log]
Waf: Entering directory /home/approx/pficommon_verify/pficommon/build' [ 1/65] cxx: src/empty.cpp -> build/src/empty.cpp.1.o [ 2/65] cxx: src/visualization/empty.cpp -> build/src/visualization/empty.cpp.1.o [ 3/65] cxx: src/system/mmapper.cpp -> build/src/system/mmapper.cpp.1.o [ 4/65] cxx: src/system/sysstat.cpp -> build/src/system/sysstat.cpp.1.o [ 5/65] cxx: src/system/time_util.cpp -> build/src/system/time_util.cpp.1.o [ 6/65] cxx: src/system/endian_util.cpp -> build/src/system/endian_util.cpp.1.o [ 7/65] cxx: src/system/syscall.cpp -> build/src/system/syscall.cpp.1.o [ 8/65] cxx: src/system/file.cpp -> build/src/system/file.cpp.1.o [ 9/65] cxx: src/network/cgi/server.cpp -> build/src/network/cgi/server.cpp.1.o [10/65] cxx: src/network/cgi/base.cpp -> build/src/network/cgi/base.cpp.1.o [11/65] cxx: src/network/cgi/util.cpp -> build/src/network/cgi/util.cpp.1.o [12/65] cxx: src/concurrent/rwmutex.cpp -> build/src/concurrent/rwmutex.cpp.1.o [13/65] cxx: src/concurrent/mutex.cpp -> build/src/concurrent/mutex.cpp.1.o [14/65] cxx: src/concurrent/internal.cpp -> build/src/concurrent/internal.cpp.1.o [15/65] cxx: src/concurrent/condition.cpp -> build/src/concurrent/condition.cpp.1.o In file included from ../src/system/file.cpp:32:0: ../src/system/file.h:53:1: error: 'ssize_t' does not name a type Waf: Leaving directory/home/approx/pficommon_verify/pficommon/build'
Build failed
-> task failed (exit status 1):
{task 90720528: cxx file.cpp -> file.cpp.1.o}
['/usr/local/bin/g++', '-O2', '-Wall', '-g', '-pipe', '-D_REENTRANT', '-fno-omit-frame-pointer', '-fPIC', '-I/home/approx/pficommon_verify/pficommon/build/src/system', '-I/home/approx/pficommon_verify/pficommon/src/system', '../src/system/file.cpp', '-c', '-o', 'src/system/file.cpp.1.o']

I cannot parse one digit character.
This code throws an exception.

  std::istringstream iss("1");
  pfi::text::json::json js;
  iss >> js;

Non finite floating values, such as NaN and inf, are not supported in JSON. But, pficommon's json library can be assigned with these values. In such case, string representations of them are invalid like this:

{"value": inf}

It is an invalid json representation.

I have three ideas for this problem:

• Reject to assign non-finite values (throw exception in constructor of json_float)
• Accept to assign these values, but reject to write them
• Accept to assign these values, and replace them on writing them (like string "nan", or null value)

unittest_gtest.py uses b'' syntax. Python 2.4 doesn't support this syntax, so waf command fails.

Python 2.4 is a legacy and official release is ended.
I think removing python 2.4 support is better.

object_stream::write returns -1 when write failed.

ssize_t rl = ::write(iofd, p, pend-p);
    if(rl <= 0) {
      if(rl == 0) {
        return -1;
      }
...

In this situation, rpc_request::write written in message.h returns true as it casts returned int value to bool.

  template <typename ObjectWritable, typename P>
  static bool write(ObjectWritable& o, uint32_t msgid, const std::string& name, const P& param, double timeout_sec)
  {
    msgpack::type::tuple<uint8_t, uint32_t, const std::string&, const P&>
      req(0, msgid, name, param);
    return o.write(req, timeout_sec); // <- HERE
  }

This problem happens in rpc_stream::send that uses rpc_request::write and object_stream. We need to check returned value of object_stream::write in rpc_request::write.

See below:

$ python3.3
Python 3.3.5 (default, Jun 30 2014, 12:20:12) 
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> __import__("unittest_gtest")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: source code string cannot contain null bytes
>>> 
$ python3.2
Python 3.2.5 (default, Jul  8 2014, 10:38:55) 
[GCC 4.8.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> __import__("unittest_gtest")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "unittest_gtest.py", line 38, in <module>
    from waflib.TaskGen import before, after, feature
ImportError: No module named waflib.TaskGen
>>> 

It seems something changed from python3.3?
e.g. minimal reproducer files foo.py library and loader.py which load foo.py

foo.py:

def foo():
  pass

# contain null byte �<replace with null byte>

loader.py:

import foo

and try to execute loader.py with several python versions

$ python3.4 loader.py
Traceback (most recent call last):
  File "loader.py", line 1, in <module>
    import foo
TypeError: source code string cannot contain null bytes
$ python3.3 loader.py
Traceback (most recent call last):
  File "loader.py", line 1, in <module>
    import foo
TypeError: source code string cannot contain null bytes
$ python3.2 loader.py
$ python2.7 loader.py

I use unordered_map in some places.
But sadly, unordered_map cannot be to_json.

Please add unordered_map to json routines.

LanetekiMacBook-Pro:pficommon Lane$ sudo ./waf configure
Password:
Setting top to : /Users/Lane/pficommon
Setting out to : /Users/Lane/pficommon/build
Checking for 'g++' (c++ compiler) : /usr/bin/g++
Unpacking gtest : yes
Checking for library pthread : yes
Checking for library pthread : yes
Checking for header msgpack.hpp : yes
Checking for header fcgiapp.h : not found
Checking for header stdint.h : yes
Checking for header unordered_map : not found
Checking for header tr1/unordered_map : yes
Checking for header ext/hash_map : yes
Checking for header unordered_set : not found
Checking for header tr1/unordered_set : yes
Checking for header ext/hash_set : yes
Checking for 'mysql_config' : not found
Checking for header postgres.h : no
Checking for 'Magick++-config' : not found

pficommon has been configured as follows:

[Modules]
FCGI module: no
Database module: yes
have MySQL lib: no
have PostgreSQL lib: no
MessagePack RPC module: yes

[Visualization]
Magick++ impl: no

[Build information]
Package: pficommon-1.3.1
build (compile on): x86_64-darwin
host endian: little
Compiler: g++
Compiler version: 4.2.1
CXXFLAGS: -O2 -Wall -g -pipe -D_REENTRANT -fno-omit-frame-pointer -D_FORTIFY_SOURCE=1

'configure' finished successfully (2.618s)

LanetekiMacBook-Pro:pficommon Lane$ sudo ./waf build
Waf: Entering directory /Users/Lane/pficommon/build' [15/72] cxx: src/network/cgi/server.cpp -> build/src/network/cgi/server.cpp.1.o [18/72] cxx: src/network/socket.cpp -> build/src/network/socket.cpp.1.o [20/72] cxx: src/network/dns.cpp -> build/src/network/dns.cpp.1.o [22/72] cxx: src/network/mprpc/object_stream.cpp -> build/src/network/mprpc/object_stream.cpp.1.o [23/72] cxx: src/network/mprpc/rpc_client.cpp -> build/src/network/mprpc/rpc_client.cpp.1.o [24/72] cxx: src/network/mprpc/rpc_server.cpp -> build/src/network/mprpc/rpc_server.cpp.1.o [25/72] cxx: src/network/mprpc/rpc_stream.cpp -> build/src/network/mprpc/rpc_stream.cpp.1.o [36/72] cxx: src/network/mprpc/socket.cpp -> build/src/network/mprpc/socket.cpp.1.o ../src/network/socket.cpp: In static member function 'static void pfi::network::stream_socket::set_dns_resolver(const pfi::lang::shared_ptr<pfi::network::dns_resolver, pfi::concurrent::threading_model::single_thread>&)': ../src/network/socket.cpp:93: error: no matching function for call to 'pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock)' ../src/network/../concurrent/lock.h:52: note: candidates are: pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock&) ../src/network/socket.cpp: In member function 'bool pfi::network::stream_socket::connect(const std::string&, uint16_t)': ../src/network/socket.cpp:109: error: no matching function for call to 'pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock)' ../src/network/../concurrent/lock.h:52: note: candidates are: pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock&) ../src/network/mprpc/socket.cpp: In static member function 'static std::vector<pfi::network::ipv4_address, std::allocator<pfi::network::ipv4_address> > pfi::network::mprpc::socket::resolve(const std::string&, uint16_t)': ../src/network/mprpc/socket.cpp:102: error: no matching function for call to 'pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock)' ../src/network/mprpc/../../concurrent/lock.h:52: note: candidates are: pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock&) ../src/network/dns.cpp: In member function 'void pfi::network::cached_dns_resolver::impl::clear_cache()': ../src/network/dns.cpp:157: error: no matching function for call to 'pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock)' ../src/network/../concurrent/lock.h:52: note: candidates are: pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock&) ../src/network/dns.cpp: In member function 'void pfi::network::cached_dns_resolver::impl::delete_cache(const std::string&, uint16_t)': ../src/network/dns.cpp:165: error: no matching function for call to 'pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock)' ../src/network/../concurrent/lock.h:52: note: candidates are: pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock&) ../src/network/dns.cpp: In member function 'std::vector<pfi::network::ipv4_address, std::allocator<pfi::network::ipv4_address> > pfi::network::cached_dns_resolver::impl::resolve(const std::string&, uint16_t)': ../src/network/dns.cpp:182: error: no matching function for call to 'pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock)' ../src/network/../concurrent/lock.h:52: note: candidates are: pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock&) ../src/network/dns.cpp:205: error: no matching function for call to 'pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock)' ../src/network/../concurrent/lock.h:52: note: candidates are: pfi::concurrent::scoped_lock::scoped_lock(pfi::concurrent::scoped_lock&) ../src/network/cgi/../../lang/bind.h:626: warning: 'pfi::lang::_4' defined but not used ../src/network/cgi/../../lang/bind.h:627: warning: 'pfi::lang::_5' defined but not used ../src/network/cgi/../../lang/bind.h:628: warning: 'pfi::lang::_6' defined but not used ../src/network/cgi/../../lang/bind.h:629: warning: 'pfi::lang::_7' defined but not used ../src/network/cgi/../../lang/bind.h:630: warning: 'pfi::lang::_8' defined but not used ../src/network/cgi/../../lang/bind.h:631: warning: 'pfi::lang::_9' defined but not used ../src/network/mprpc/../../lang/bind.h:626: warning: 'pfi::lang::_4' defined but not used ../src/network/mprpc/../../lang/bind.h:627: warning: 'pfi::lang::_5' defined but not used ../src/network/mprpc/../../lang/bind.h:628: warning: 'pfi::lang::_6' defined but not used ../src/network/mprpc/../../lang/bind.h:629: warning: 'pfi::lang::_7' defined but not used ../src/network/mprpc/../../lang/bind.h:630: warning: 'pfi::lang::_8' defined but not used ../src/network/mprpc/../../lang/bind.h:631: warning: 'pfi::lang::_9' defined but not used Waf: Leaving directory/Users/Lane/pficommon/build'
Build failed
-> task in 'pficommon_network_base' failed (exit status 1):
{task 4393821520: cxx socket.cpp -> socket.cpp.1.o}
['/usr/bin/g++', '-O2', '-Wall', '-g', '-pipe', '-D_REENTRANT', '-fno-omit-frame-pointer', '-D_FORTIFY_SOURCE=1', '-fPIC', '-compatibility_version', '1', '-current_version', '1', '-I/Users/Lane/pficommon/build/src/network', '-I/Users/Lane/pficommon/src/network', '../src/network/socket.cpp', '-c', '-o', 'src/network/socket.cpp.1.o']
-> task in 'pficommon_network_mprpc' failed (exit status 1):
{task 4393858512: cxx socket.cpp -> socket.cpp.1.o}
['/usr/bin/g++', '-O2', '-Wall', '-g', '-pipe', '-D_REENTRANT', '-fno-omit-frame-pointer', '-D_FORTIFY_SOURCE=1', '-fPIC', '-compatibility_version', '1', '-current_version', '1', '../src/network/mprpc/socket.cpp', '-c', '-o', 'src/network/mprpc/socket.cpp.1.o']
-> task in 'pficommon_network_base' failed (exit status 1):
{task 4393821776: cxx dns.cpp -> dns.cpp.1.o}
['/usr/bin/g++', '-O2', '-Wall', '-g', '-pipe', '-D_REENTRANT', '-fno-omit-frame-pointer', '-D_FORTIFY_SOURCE=1', '-fPIC', '-compatibility_version', '1', '-current_version', '1', '-I/Users/Lane/pficommon/build/src/network', '-I/Users/Lane/pficommon/src/network', '../src/network/dns.cpp', '-c', '-o', 'src/network/dns.cpp.1.o']
LanetekiMacBook-Pro:pficommon Lane$

I installed the pficommon successfully before, I re-install my OS yesterday and find this issue, the configure shows the essential libs are ok.
Do I miss something or wrong configure on my OS?

I have encountered failure of pficommon's installation occasionally.

$ ./waf install
Waf: Entering directory `/home/eiichiro/git/pficommon/build'
+ install /home/eiichiro/local/include/pficommon/pfi-config.h (from build/src/pfi-config.h)
+ install /home/eiichiro/local/include/pficommon/lang/cast.h (from src/lang/cast.h)
.
.
.
+ install /home/eiichiro/local/include/pficommon/data/functional_hash.h (from src/data/functional_hash.h)
+ install /home/eiichiro/local/include/pficommon/data/intern.h (from src/data/intern.h)
Waf: Leaving directory `/home/eiichiro/git/pficommon/build'
Build failed
Traceback (most recent call last):
  File "/home/eiichiro/git/pficommon/.waf-1.6.7-0a94702c61504c487a251b8d0a04ca9a/waflib/Task.py", line 125, in process
    ret=self.run()
  File "/home/eiichiro/git/pficommon/.waf-1.6.7-0a94702c61504c487a251b8d0a04ca9a/waflib/Task.py", line 49, in run
    return m1(self)
  File "/home/eiichiro/git/pficommon/.waf-1.6.7-0a94702c61504c487a251b8d0a04ca9a/waflib/Build.py", line 444, in run
    return self.generator.exec_task()
  File "/home/eiichiro/git/pficommon/.waf-1.6.7-0a94702c61504c487a251b8d0a04ca9a/waflib/Build.py", line 461, in exec_install_files
    self.generator.bld.do_install(y.abspath(),destfile,self.chmod)
  File "/home/eiichiro/git/pficommon/.waf-1.6.7-0a94702c61504c487a251b8d0a04ca9a/waflib/Build.py", line 499, in do_install
    shutil.copy2(src,tgt)
  File "/usr/lib64/python2.6/shutil.py", line 96, in copy2
    copystat(src, dst)
  File "/usr/lib64/python2.6/shutil.py", line 66, in copystat
    os.utime(dst, (st.st_atime, st.st_mtime))
OSError: [Errno 2] No such file or directory: '/home/eiichiro/local/lib/pkgconfig/pficommon.pc'

And found two ways to work around this behavior.

• install with -j1 option(./waf install -j1)
• install TWICE(./waf install || ./waf install)

The normal installation(./waf install) succeeded with a probability of about 78% (788 / 1000).
But, when using either of above two work arounds, all installations succeeded (1000 / 1000).

I think that installation tasks may have dependencies and waf execute those tasks concurrently.

I'm looking for solutions to this problem...

• Can I avoid this problem by fixing the wscript? (write those dependencies explicitly in the wcript)
• Can I avoid this problem by updating the waf? (Is this the bug of the waf?)

Please tell me if there is any good solution.

I would like to pack pficommon for Debian.
Could you tag the pficommon that include naota344's patch?

Best regards,
Kiwamu Okabe

Now, json_number is an orphan class ;(

When initialization of CGIs is failed (for example when clone method throws an exception at https://github.com/pfi/pficommon/blob/master/src/network/cgi/server.cpp#L168 ), run_server::ths is not initialized correctly. But, on run_server::~run_server it try to join threads at https://github.com/pfi/pficommon/blob/master/src/network/cgi/server.cpp#L136 , and it causes segfault as ths[i] is null at https://github.com/pfi/pficommon/blob/master/src/network/cgi/server.cpp#L184 .

• Jubatus pj has same problem: jubatus/jubatus#102

stream=shared_ptr<iostream>(new basic_httpbody_stream<char>(sock, lexical_cast<int>(header_["Content-Length"])));

lexical_cast raises an exception when the value in Content-Length is greater than 2^31-1.

basic_httpbody_stream also stores a remaining file size as int. It should also be fixed.

bool exist_id(int id) const {
    return id<(int)lbt.size();
}

This method should return false for a negative id.
pfi::data::intern generates a non-negative id for any key.

We should fix tests that assume the hash key ordering.

$ ./waf --checkone=json_test
Waf: Entering directory `/home/eiichiro/git/pficommon/build'
[70/78] utest: build/src/text/json_test
Waf: Leaving directory `/home/eiichiro/git/pficommon/build'
test summary
  tests that pass 0/1
  tests that fail 1/1
    /home/eiichiro/git/pficommon/build/src/text/json_test
Running main() from gtest_main.cc
[==========] Running 27 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 27 tests from json
[ RUN      ] json.to_json
../src/text/json_test.cpp:240: Failure
Value of: oss.str()
  Actual: "{\"hoge\":3.14,\"abc\":1.23}"
Expected: "{\"abc\":1.23,\"hoge\":3.14}"
Which is: "{"abc":1.23,"hoge":3.14}"
[  FAILED  ] json.to_json (1 ms)
[ RUN      ] json.from_json
[       OK ] json.from_json (0 ms)
[ RUN      ] json.merge
[       OK ] json.merge (0 ms)
[ RUN      ] json.merge_with
[       OK ] json.merge_with (0 ms)
[ RUN      ] json.size
[       OK ] json.size (0 ms)
[ RUN      ] json.type
[       OK ] json.type (0 ms)
[ RUN      ] json.is
[       OK ] json.is (0 ms)
[ RUN      ] json.parse
[       OK ] json.parse (0 ms)
[ RUN      ] json.bignum_parse
[       OK ] json.bignum_parse (0 ms)
[ RUN      ] json.pretty
[       OK ] json.pretty (0 ms)
[ RUN      ] json.clone
[       OK ] json.clone (0 ms)
[ RUN      ] json.utf8
[       OK ] json.utf8 (0 ms)
[ RUN      ] json.utf8_2
[       OK ] json.utf8_2 (0 ms)
[ RUN      ] json.utf8_without_escaping
[       OK ] json.utf8_without_escaping (0 ms)
[ RUN      ] json.with_default
[       OK ] json.with_default (0 ms)
[ RUN      ] json.optional
../src/text/json_test.cpp:1016: Failure
Value of: "{\"abc\":123,\"def\":null}"
  Actual: "{"abc":123,"def":null}"
Expected: oss.str()
Which is: "{\"def\":null,\"abc\":123}"
../src/text/json_test.cpp:1023: Failure
Value of: "{\"abc\":123,\"def\":456}"
  Actual: "{"abc":123,"def":456}"
Expected: oss.str()
Which is: "{\"def\":456,\"abc\":123}"
[  FAILED  ] json.optional (0 ms)
[ RUN      ] json.cast_to_json
[       OK ] json.cast_to_json (0 ms)
[ RUN      ] json.serialization
[       OK ] json.serialization (1 ms)
[ RUN      ] json.out_of_range
[       OK ] json.out_of_range (0 ms)
[ RUN      ] json.parse_multiple_jsons
[       OK ] json.parse_multiple_jsons (0 ms)
[ RUN      ] json.parse_invalid_multiple_json
[       OK ] json.parse_invalid_multiple_json (0 ms)
[ RUN      ] json.escape_ctrl
[       OK ] json.escape_ctrl (0 ms)
[ RUN      ] json.invalid_json_num
[       OK ] json.invalid_json_num (0 ms)
[ RUN      ] json.finite
[       OK ] json.finite (0 ms)
[ RUN      ] json.named_optional_value
[       OK ] json.named_optional_value (0 ms)
[ RUN      ] json.named_optional_value_null
[       OK ] json.named_optional_value_null (0 ms)
[ RUN      ] json.not_found_member
[       OK ] json.not_found_member (0 ms)
[----------] 27 tests from json (2 ms total)

[----------] Global test environment tear-down
[==========] 27 tests from 1 test case ran. (2 ms total)
[  PASSED  ] 25 tests.
[  FAILED  ] 2 tests, listed below:
[  FAILED  ] json.to_json
[  FAILED  ] json.optional

 2 FAILED TESTS

test failed

Is pfi::data::string::sanitize_utf8 implementation correct behavior? That removes UTF-8 control characters.

UTF-8 has control-character 0x00 to 0x1F(U+0000 to U+001F), see also http://www.utf8-chartable.de

src/network/socket.cpp compilation failure because pfi::lang was missing by #86
Please check #86 replacement const shared_ptr.

https://github.com/pfi/pficommon/blob/master/src/network/mprpc/caller.h#L51
https://github.com/pfi/pficommon/blob/master/src/network/rpc/caller.h#L48

These functions should be moved to detail namespace.

Replace newer unittest_gtest.py.

NOTE: jubatus/jubatus#296 depends on pficommon.

data/sparse_matrix_test and text/json_test fail.

https://github.com/pfi/pficommon/blob/master/src/data/suffix_array/suffix_array.h#L35

JSON desirialization doesn't throw json_bad_cast when desirializing struct with array.
Modified test outputs like this.

[ RUN      ] json.parse
unknown file: Failure
C++ exception with description "failed to use json as object." thrown in the test body.
[  FAILED  ] json.parse (0 ms)

diff --git a/src/text/json_test.cpp b/src/text/json_test.cpp
index acd308b..b1fc453 100644
--- a/src/text/json_test.cpp
+++ b/src/text/json_test.cpp
@@ -691,11 +691,8 @@ TEST(json, parse)
       "\"Width\":  800,\n"
       "\"Height\": 600,\n"
       "\"Title\":  \"View from 15th Floor\",\n"
-      "\"Thumbnail\": {\n"
-      "\"Url\":    \"http://www.example.com/image/481989943\",\n"
-      "\"Height\": 125,\n"
-      "\"Width\":  \"100\"\n"
-      "},\n"
+      "\"Thumbnail\": []\n"
+      ",\n"
       "\"IDs\": [116, 943, 234, 38793]\n"
       "}\n"
       "}"

In Jubatus project, the bug was discovered.
jubatus/jubatus#773

Currently, we use dynamic_cast to check the json type.
Adding type information to json classes makes checking more simple.

Since json allow only 4 hexes after \u, should use UTF-16 surrogate pair for non Basic Multilingual Plane charctors.

https://tools.ietf.org/html/rfc4627#section-2.5

To escape an extended character that is not in the Basic Multilingual
Plane, the character is represented as a twelve-character sequence,
encoding the UTF-16 surrogate pair. So, for example, a string
containing only the G clef character (U+1D11E) may be represented as
"\uD834\uDD1E".

$ cat test.cc
#include <pficommon/text/json.h>
#include <iostream>

using namespace pfi::text::json;
int main() {
  std::cout << json(new json_string("\U0001D11E")) << std::endl;
}
$ ./a.out
"\uD11E"

https://github.com/retrieva/pficommon/blob/master/src/text/json/base.h#L341-L347

Jubatus's server-client tests often fails because unittests starts before test-server complete listen(2)

ref: jubatus/jubatus#35

When I cast json object to a user-defined struct and a missing variable is found, I got an unreadable error message.

For example, I wrote this code.

struct T {
  int n;
  template <typename Ar>
  void serialize(Ar& ar) {
    ar & MEMBER(n);
  }
};

int main() {
  pfi::text::json::json js(new pfi::text::json::json_object);  // no field
  T t = pfi::text::json::json_cast<T>(js);
}

This causes json_bad_cast, but its error message is unreadable:

terminate called after throwing an instance of 'pfi::text::json::json_bad_cast<long>'
  what():  Failed json_cast from N3pfi4text4json9json_nullE to l.

It is because json_cast try to convert null to named_value when no field is found:
https://github.com/pfi/pficommon/blob/master/src/text/json/cast.h#L537
Usually null cannot be converted to a user-defined member variable. So, it causes such an error message.

It works well, when a member variable is named_value<optional<T> >, because optional ignores null value:
https://github.com/pfi/pficommon/blob/master/src/text/json/cast.h#L554

In my opinion, when an expected variable is not found, we need to throw an error that reports the name of the variable. And, when a member variable is optional, we need to ignore it.
This is an implementation of my library which works so:
https://github.com/unnonouno/jsonconfig/blob/master/src/jsonconfig/cast.hpp#L161
https://github.com/unnonouno/jsonconfig/blob/master/src/jsonconfig/cast.hpp#L172

http://pfi.github.com/pficommon/network/iostream.html

socketstream's constructor which takes address needs a port number as following:

socketstream ss("www.google.com", 80);

These functions aren't described in document.

• scoped_rlock
• scoped_wrlock

https://github.com/pfi/pficommon/blob/master/src/network/mprpc/caller.h#L63
https://github.com/pfi/pficommon/blob/master/src/network/rpc/caller.h#L66

pficommon may want to use only space but some code are using tabs.
http://www.emacswiki.org/emacs/TabsSpacesBoth

Why use std::map internally?
unordered_map is better than std::map for improving speed and memory usage.

pficommon depends pthread but pc doesn't have -lpthread flag.

JavaScript object doesn't have length property.
But other languages support size/length (or keys property) for AssociativeArray.

So, I want size support for json_object.
Currently, I use distance(json.begin(), json.end()) to calculate the size of json_object.

pfi::text::json::json stores a content using shared_ptr.
So, json1 = json2; is pointer copy.
I want clone to copy a content.