wavii / darner Goto Github PK

View Code? Open in Web Editor NEW

866.0 866.0 79.0 761 KB

simple, lightweight message queue

License: Other

C++ 91.30% Shell 5.26% Objective-C 3.44%

darner's People

Contributors

Stargazers

Watchers

darner's Issues

More helpful exceptions

Mainly user error, but exceptions (when starting up) are a bit un-helpful. Here is an example of a invalid data directory on OSX.

[INFO] 2012-Sep-22 17:24:47.901668: darner: queue server
[INFO] 2012-Sep-22 17:24:47.902440: build: Aug 30 2012 (17:15:58) v0.1.4 (c) Wavii, Inc.
[INFO] 2012-Sep-22 17:24:47.902481: listening on port: 22133
[INFO] 2012-Sep-22 17:24:47.902515: data dir: /tmp/
[INFO] 2012-Sep-22 17:24:47.902544: debug logging is turned ON
[INFO] 2012-Sep-22 17:24:47.902574: starting up
libc++abi.dylib: terminate called throwing an exception
Abort trap: 6

Can't build it on Mavericks (Mac OS X 10.9)

Hi,

Is the project alive? Can't build it on Mavericks with Xcode nor gcc-4.2.

Documentation Problems

"Erik Frey", "Homebrew", and two "Kestrel" links are broken.

Also, under "Resident Memory", there is one errant line that says "Kestrel", which, I believe should be "Darner". It doesn't look good:

kestrel   262024 requests: 465148 kB
kestrel   524048 requests: 520476 kB
kestrel  1048576 requests: 611612 kB
darner         0 requests: 2220 kB
darner      1024 requests: 3492 kB
darner      2048 requests: 5872 kB
darner      4096 requests: 8136 kB
darner      8192 requests: 15520 kB
darner     16384 requests: 25656 kB
darner     32768 requests: 27412 kB
darner     65536 requests: 24324 kB
darner    131072 requests: 28440 kB
darner    262024 requests: 28524 kB
darner    524048 requests: 32104 kB
kestrel  1048576 requests: 33848 kB

Stats output.

Kestrel have a simple webinterface called stats.json which output some statistics. Would be cool if darner had this aswell.

possible queue corruption on upgrade. anything to look for?

I think I have some queue corruption. We upgraded 0.1.4 to 0.2.0 via put binary in place, stop; start; of the same dir.

After restart, the clients of one queue started getting:

SERVER_ERROR Input/output error

I copied off the spool dir and ran darner on it in a separate environment to see what the state looked like. I got a few of these in the darner output:

[ERROR] 2013-Jul-11 00:25:54.528810: handler<0x7fe7d00047d0>::get: Input/output error

Also odd, one queue in stats shows almost 42 million items in it, but before the upgrade it had very few. During the upgrade the open transactions stat also showed what appeared to be a wrapped 64bit int (it was huge, sadly I didn't record it).

I am poking at things a bit more.

Anything in particular to look for?
The darner instance was under pretty heavy load when it was restarted, I am wondering if the older version didn't exit gracefully and may have corrupted the leveldb transaction log or something.

Configuring multi-server darner setup

Could you please provide documentation or atleast a sample config that shows how we can setup a multi-server darner queue cluster.

Odd behavior with /open and later /close

I am getting some odd behavior with doing an /open, performing work, and a subsequent /close.

If I use a single worker client, it performs as expected. When I use more than one connection (python with greenlets) I sometimes get the expected behavior (one client "/open"s the message, appears to get a lock on it, then "/close"s it), and sometimes each client gets the job multiple times, and it seems the close it ignored (or perhaps the message isn't being locked).

Darner is in debug log mode, but no logging is occurring, so it is hard to tell what is doing on inside the server. Dumping the memcache protocol shows that client side source ports are not changing (implies that connections are indeed persistent and not churning).

Any advice on what to look for?

Thanks.

Notes:

Using boost 1.50.0
darner from current master/HEAD

core dump with invalid options

$ darner -V
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::program_options::unknown_option> >'
  what():  unrecognised option '-V'
Aborted (core dumped)

darner version is 0.2.0.
boost version is 1.50.0

/close returns an object and closes the transaction

I was trying to /close a message but not get the next one yet.

stats
STAT queue_job_queue_items 2
STAT queue_job_queue_waiters 0
STAT queue_job_queue_open_transactions 0
END

get job_queue/open
VALUE job_queue 0 32
{"url": "http://www.google.com"}
END

stats
STAT queue_job_queue_items 1
STAT queue_job_queue_waiters 0
STAT queue_job_queue_open_transactions 1
END

get job_queue/close
VALUE job_queue 0 32
{"url": "http://www.google.com"}
END

stats
STAT queue_job_queue_items 0
STAT queue_job_queue_waiters 0
STAT queue_job_queue_open_transactions 0
END

It looks like /close is returning a message, as well as closing the transaction (and not opening a new transaction like /close/open would).

Is this expected behavior?

Can't build on windows

Hi,

Can't build it on windows platform? Can you provide any guidance for building this project on windows?Do you have any plans to support it in future?

Talking to darner from ruby memcache gems

I've tried kestrel-client (build on top of memcached gem), the memcached gem and the dalli gem, and all fail to perform a get from darner.

For example, I start up darner:

$ darner -d /var/spool/darner/
[INFO] 2012-Sep-07 19:41:46.008201: darner: queue server
[INFO] 2012-Sep-07 19:41:46.008847: build: Sep  7 2012 (18:54:17) v0.1.4 (c) Wavii, Inc.
[INFO] 2012-Sep-07 19:41:46.008932: listening on port: 22133
[INFO] 2012-Sep-07 19:41:46.008978: data dir: /var/spool/darner/
[INFO] 2012-Sep-07 19:41:46.009047: starting up

Then start up irb, with dalli:

$ irb -r rubygems -r dalli
1.9.3p0 :001 > client = Dalli::Client.new 'localhost:22133'
 => #<Dalli::Client:0x007fae532e3f20 @servers="localhost:22133", @options={}, @ring=nil> 
1.9.3p0 :002 > client.get 'foo'
I, [2012-09-07T19:44:05.305352 #77838]  INFO -- : localhost:22133 failed (count: 0)
Dalli::RingError: No server available
    from /Users/jnunemaker/.rvm/gems/ruby-1.9.3-p0/gems/dalli-2.1.0/lib/dalli/ring.rb:45:in `server_for_key'
    from /Users/jnunemaker/.rvm/gems/ruby-1.9.3-p0/gems/dalli-2.1.0/lib/dalli/client.rb:250:in `perform'
    from /Users/jnunemaker/.rvm/gems/ruby-1.9.3-p0/gems/dalli-2.1.0/lib/dalli/client.rb:50:in `get'
    from (irb):2
    from /Users/jnunemaker/.rvm/rubies/ruby-1.9.3-p0/bin/irb:16:in `<main>'

Same thing from the memcached gem:

$ irb -r rubygems -r memcached
1.9.3p0 :001 > client = Memcached.new 'localhost:22133'
 => #<Memcached:0x007fe0430bc838 @struct=#<Rlibmemcached::MemcachedSt:0x007fe0430bc7e8>, @options={:hash=>:fnv1_32, :no_block=>false, :noreply=>false, :distribution=>:consistent_ketama, :ketama_weighted=>true, :buffer_requests=>false, :cache_lookups=>true, :support_cas=>false, :tcp_nodelay=>false, :show_backtraces=>false, :retry_timeout=>30, :timeout=>0.25, :rcv_timeout=>0.25, :poll_timeout=>0.25, :connect_timeout=>4, :prefix_key=>"", :prefix_delimiter=>"", :hash_with_prefix_key=>true, :default_ttl=>604800, :default_weight=>8, :sort_hosts=>false, :auto_eject_hosts=>true, :server_failure_limit=>2, :verify_key=>true, :use_udp=>false, :binary_protocol=>false, :credentials=>nil, :experimental_features=>false, :exception_retry_limit=>5, :exceptions_to_retry=>[Memcached::ServerIsMarkedDead, Memcached::ATimeoutOccurred, Memcached::ConnectionBindFailure, Memcached::ConnectionFailure, Memcached::ConnectionSocketCreateFailure, Memcached::Failure, Memcached::MemoryAllocationFailure, Memcached::ReadFailure, Memcached::ServerEnd, Memcached::ServerError, Memcached::SystemError, Memcached::UnknownReadFailure, Memcached::WriteFailure, Memcached::SomeErrorsWereReported]}, @default_ttl=604800, @not_found=#<Memcached::NotFound: Memcached::NotFound>, @not_stored=#<Memcached::NotStored: Memcached::NotStored>> 
1.9.3p0 :002 > client.get 'foo'
return else
"foo"
8
Memcached::ProtocolError: Key {"foo"=>"localhost:22133:8"}
    from /Users/jnunemaker/.rvm/gems/ruby-1.9.3-p0/gems/memcached-1.4.3/lib/memcached/memcached.rb:623:in `reraise'
    from /Users/jnunemaker/.rvm/gems/ruby-1.9.3-p0/gems/memcached-1.4.3/lib/memcached/memcached.rb:600:in `check_return_code'
    from /Users/jnunemaker/.rvm/gems/ruby-1.9.3-p0/gems/memcached-1.4.3/lib/memcached/memcached.rb:517:in `get'
    from (irb):2
    from /Users/jnunemaker/.rvm/rubies/ruby-1.9.3-p0/bin/irb:16:in `<main>'

set seems to work for the memcached gem, but dalli always reports darner as dead. I also tried with the raw flag true, to avoid marshaling, but it still failed.

I was able to use telnet just fine, so it seems to be something specific that the ruby gems need in the memcache protocol, but I'm not familiar enough. I can try to dig in, but I thought I would see if it is something obvious or easy that I am missing first.

I currently use kestrel with the kestrel-client gem no problem. I'd love to test out darner. Any ideas?

Error building. Boost too old?

When trying to build on centos 6.3, I get this error:

$ make
[  6%] Building CXX object CMakeFiles/darner.dir/src/net/handler.o
In file included from /build/darner/include/darner/queue/iqstream.h:6,
                 from /build/darner/include/darner/net/handler.h:14,
                 from /build/darner/src/net/handler.cpp:1:
/build/darner/include/darner/queue/queue.h: In member function ‘void darner::queue::put(const darner::queue::key_type&, const std::string&)’:
/build/darner/include/darner/queue/queue.h:211: error: no match for call to ‘(const boost::system::error_category) ()’
/build/darner/include/darner/queue/queue.h: In member function ‘void darner::queue::get(const darner::queue::key_type&, std::string&)’:
/build/darner/include/darner/queue/queue.h:217: error: no match for call to ‘(const boost::system::error_category) ()’
/build/darner/include/darner/queue/queue.h: In member function ‘void darner::queue::write(leveldb::WriteBatch&)’:
/build/darner/include/darner/queue/queue.h:223: error: no match for call to ‘(const boost::system::error_category) ()’
In file included from /build/darner/include/darner/net/handler.h:16,
                 from /build/darner/src/net/handler.cpp:1:
/build/darner/include/darner/util/queue_map.hpp: In constructor ‘darner::queue_map::queue_map(boost::asio::io_service&, const std::string&)’:
/build/darner/include/darner/util/queue_map.hpp:33: error: ‘struct std::basic_string<char, std::char_traits<char>, std::allocator<char> >’ has no member named ‘string’
make[2]: *** [CMakeFiles/darner.dir/src/net/handler.o] Error 1
make[1]: *** [CMakeFiles/darner.dir/all] Error 2
make: *** [all] Error 2

Boost version is 1.41.0 (version in centos repos).
Is this version too old?

support memcached binary protocol

Darner ought to support the memcached binary protocol:

It's simpler and faster to parse.
Some newer memcached libraries, such as dalli, support the binary protocol exclusively.

ERROR on quit

If I connect to darner via telnet and quit, I'll get an error:

$ telnet localhost 22133
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
quit
ERROR 
Connection closed by foreign host.

Add readme install for homebrew

if Homebrew/legacy-homebrew#14203 gets merged it would be nice to have instructions for homebrew.

Failing build

make 
Scanning dependencies of target darner
Scanning dependencies of target db
Scanning dependencies of target test
[  6%] [ 18%] [ 25%] [ 25%] [ 31%] [ 37%] [ 43%] [ 50%] Building CXX object CMakeFiles/db.dir/bench/db.o
Building CXX object CMakeFiles/test.dir/src/net/request.o
Building CXX object CMakeFiles/darner.dir/src/net/handler.o
Building CXX object CMakeFiles/darner.dir/src/net/request.o
Building CXX object CMakeFiles/darner.dir/src/queue/iqstream.o
Building CXX object CMakeFiles/test.dir/src/queue/iqstream.o
Building CXX object CMakeFiles/darner.dir/src/util/log.o
Building CXX object CMakeFiles/test.dir/src/queue/oqstream.o
[ 56%] [ 62%] Building CXX object CMakeFiles/test.dir/src/queue/queue.o
Building CXX object CMakeFiles/darner.dir/src/queue/oqstream.o
[ 68%] Building CXX object CMakeFiles/darner.dir/src/queue/queue.o
[ 75%] Building CXX object CMakeFiles/test.dir/src/util/log.o
In file included from /tmp/homebrew-darner-0.1.0-67cD/wavii-darner-7f087e5/src/net/handler.cpp:1:
/tmp/homebrew-darner-0.1.0-67cD/wavii-darner-7f087e5/include/darner/net/handler.h:83:19: error: no matching function for call to 'buffer'
         socket_, boost::asio::buffer(msg), boost::bind(&handler::read_request, shared_from_this(), _1, _2));
                  ^~~~~~~~~~~~~~~~~~~
/usr/local/include/boost/asio/buffer.hpp:698:26: note: candidate function not viable: no known conversion from 'const char *' to 'const boost::asio::mutable_buffer' for 1st argument;
inline mutable_buffers_1 buffer(const mutable_buffer& b)
                         ^
/usr/local/include/boost/asio/buffer.hpp:727:24: note: candidate function not viable: no known conversion from 'const char *' to 'const boost::asio::const_buffer' for 1st argument;
inline const_buffers_1 buffer(const const_buffer& b)
                       ^
/usr/local/include/boost/asio/buffer.hpp:779:26: note: candidate template ignored: failed template argument deduction
inline mutable_buffers_1 buffer(PodType (&data)[N])
                         ^
/usr/local/include/boost/asio/buffer.hpp:809:24: note: candidate template ignored: failed template argument deduction
inline const_buffers_1 buffer(const PodType (&data)[N])
                       ^
/usr/local/include/boost/asio/buffer.hpp:910:26: note: candidate template ignored: failed template argument deduction
inline mutable_buffers_1 buffer(boost::array<PodType, N>& data)
                         ^
/usr/local/include/boost/asio/buffer.hpp:941:24: note: candidate template ignored: failed template argument deduction
inline const_buffers_1 buffer(boost::array<const PodType, N>& data)
                       ^
/usr/local/include/boost/asio/buffer.hpp:975:24: note: candidate template ignored: failed template argument deduction
inline const_buffers_1 buffer(const boost::array<PodType, N>& data)
                       ^
/usr/local/include/boost/asio/buffer.hpp:1106:26: note: candidate template ignored: failed template argument deduction
inline mutable_buffers_1 buffer(std::vector<PodType, Allocator>& data)
                         ^
/usr/local/include/boost/asio/buffer.hpp:1155:24: note: candidate template ignored: failed template argument deduction
inline const_buffers_1 buffer(
                       ^
/usr/local/include/boost/asio/buffer.hpp:1202:24: note: candidate template ignored: failed template argument deduction
inline const_buffers_1 buffer(
                       ^
/usr/local/include/boost/asio/buffer.hpp:792:26: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline mutable_buffers_1 buffer(PodType (&data)[N],
                         ^
/usr/local/include/boost/asio/buffer.hpp:822:24: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(const PodType (&data)[N],
                       ^
/usr/local/include/boost/asio/buffer.hpp:924:26: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline mutable_buffers_1 buffer(boost::array<PodType, N>& data,
                         ^
/usr/local/include/boost/asio/buffer.hpp:955:24: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(boost::array<const PodType, N>& data,
                       ^
/usr/local/include/boost/asio/buffer.hpp:989:24: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(const boost::array<PodType, N>& data,
                       ^
/usr/local/include/boost/asio/buffer.hpp:1129:26: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline mutable_buffers_1 buffer(std::vector<PodType, Allocator>& data,
                         ^
/usr/local/include/boost/asio/buffer.hpp:1179:24: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(
                       ^
/usr/local/include/boost/asio/buffer.hpp:1225:24: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(
                       ^
/usr/local/include/boost/asio/buffer.hpp:765:24: note: candidate function not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(const void* data,
                       ^
/usr/local/include/boost/asio/buffer.hpp:756:26: note: candidate function not viable: requires 2 arguments, but 1 was provided
inline mutable_buffers_1 buffer(void* data, std::size_t size_in_bytes)
                         ^
/usr/local/include/boost/asio/buffer.hpp:739:24: note: candidate function not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(const const_buffer& b,
                       ^
/usr/local/include/boost/asio/buffer.hpp:710:26: note: candidate function not viable: requires 2 arguments, but 1 was provided
inline mutable_buffers_1 buffer(const mutable_buffer& b,
                         ^
1 error generated.
make[2]: *** [CMakeFiles/darner.dir/src/net/handler.o] Error 1
make[2]: *** Waiting for unfinished jobs....
[ 81%] Building CXX object CMakeFiles/test.dir/tests/queue.o
[ 87%] Building CXX object CMakeFiles/test.dir/tests/request.o
[ 93%] Building CXX object CMakeFiles/test.dir/tests/main.o
Linking CXX executable db
[ 93%] Built target db
make[1]: *** [CMakeFiles/darner.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Linking CXX executable test
[ 93%] Built target test
make: *** [all] Error 2
Error: #<BuildError: Failed executing: make >

wavii / darner Goto Github PK

darner's People

Contributors

Stargazers

Watchers

Forkers

darner's Issues

Recommend Projects

Recommend Topics

Recommend Org