Giter VIP home page Giter VIP logo

memcached's Introduction

Memcached

Memcached is a high performance multithreaded event-based key/value cache store intended to be used in a distributed system.

See: https://memcached.org/about

A fun story explaining usage: https://memcached.org/tutorial

If you're having trouble, try the wiki: https://memcached.org/wiki

If you're trying to troubleshoot odd behavior or timeouts, see: https://memcached.org/timeouts

https://memcached.org/ is a good resource in general. Please use the mailing list to ask questions, github issues aren't seen by everyone!

Dependencies

  • libevent - https://www.monkey.org/~provos/libevent/ (libevent-dev)
  • libseccomp (optional, experimental, linux) - enables process restrictions for better security. Tested only on x86-64 architectures.
  • openssl (optional) - enables TLS support. need relatively up to date version. pkg-config is needed to find openssl dependencies (such as -lz).

Environment

Be warned that the -k (mlockall) option to memcached might be dangerous when using a large cache. Just make sure the memcached machines don't swap. memcached does non-blocking network I/O, but not disk. (it should never go to disk, or you've lost the whole point of it)

Build status

See https://build.memcached.org/ for multi-platform regression testing status.

Bug reports

Feel free to use the issue tracker on github.

If you are reporting a security bug please contact a maintainer privately. We follow responsible disclosure: we handle reports privately, prepare a patch, allow notifications to vendor lists. Then we push a fix release and your bug can be posted publicly with credit in our release notes and commit history.

Website

Contributing

See https://github.com/memcached/memcached/wiki/DevelopmentRepos

memcached's People

Contributors

bradfitz avatar brianaker avatar candrews avatar danielschemmel avatar devnexen avatar dormando avatar dustin avatar elfchief avatar evmar avatar feihu-stripe avatar ffontaine avatar finnitoproductions avatar guikcd avatar ingenthr avatar kroki avatar lindner avatar linkiwi avatar minkikim89 avatar mlichvar avatar olajep avatar sgrimm avatar smukil avatar steveyen avatar swills avatar tharanga avatar thatsafunnyname avatar tomaskorbar avatar toofishes avatar trondn avatar viraptor avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

memcached's Issues

a lot of hv parameter in fact take no sense

do_item_link(item, hv);
do_item_unlink(item, hv);
do_item_unlink_nolock(item, hv);
assoc_delete(ITEM_key(it), it->nkey, hv);

and other functions so on, the hv parameter are not actually used, even in low-level
assoc_insert(it, hv); the hv can also been calculated by param it!

Evictions rate increased 3x after enabling slab_automove and lru_maintainer at the same time

Update:

TL;DR: This issue can be reproduced at least in 1.4.24 and was resolved in 1.4.25 .


I just upgraded a few memcached instances from version 1.4.5 to version 1.4.24, it works fine, but if I enable both slab_automove and lru_maintainer features (-o slab_reassign,slab_automove,lru_crawler,lru_maintainer) at the same time, the evictions rate increases from N to 4 x N(and doesn't not increase anymore):

screenshot 2016-01-18 11 16 09

(the instance was started on 1/8)

I also tried a few permutations of these arguments:

(no image)
without -o: works fine (can not reproduce).


screenshot 2016-01-18 11 29 29

with `-o lru_crawler`: works fine(the instance was started on 1/13).

screenshot 2016-01-18 11 30 48

with -o lru_crawler,lru_maintainer: works fine.(the instance was started on 1/13)


screenshot 2016-01-18 11 32 38

with -o slab_reassign: works fine.(the instance was started on 1/15)


screenshot 2016-01-18 11 33 41

with -o slab_reassign,slab_automove: works fine.(the instance was started on 1/15)


I noticed that the slab_automove is an experimental feature firstly introduced in version 1.4.11 and refactored in version 1.4.25 while not listed in the output memcached -h of version 1.4.24. Please close the issue if it's a known issue or already fixed in #113 .

Where to ask questions on Memcache?

This is a Github Issue tracker. But I find that I have some questions relaed to Memcache which I wanted to ask. Is there a forumn dedicated to Memcached or is it fine to ask it here and it could be tagged as a FAQ? Can anyone please help on this.

Clarify expected behaviour for negative expiration times

I'm trying to find out the expected behaviour when a negative expiry time is sent with a command.

Django seems sure that a negative value passed to the client should mean immediately expire:
https://github.com/django/django/blob/1.9.1/django/core/cache/backends/memcached.py#L50-L53
...and pylibmc's docs hint at expected behaviour, but aren't overly clear:
http://sendapatch.se/projects/pylibmc/misc.html#differences-from-python-memcached

However the protocol docs don't currently mention negative values:

Expiration times
----------------
Some commands involve a client sending some kind of expiration time
(relative to an item or to an operation requested by the client) to
the server. In all such cases, the actual value sent may either be
Unix time (number of seconds since January 1, 1970, as a 32-bit
value), or a number of seconds starting from current time. In the
latter case, this number of seconds may not exceed 60*60*24*30 (number
of seconds in 30 days); if the number sent by a client is larger than
that, the server will consider it to be real Unix time value rather
than an offset from current time.

memcached/doc/protocol.txt

Lines 163 to 168 in 6b65e45

- <exptime> is expiration time. If it's 0, the item never expires
(although it may be deleted from the cache to make place for other
items). If it's non-zero (either Unix time or offset in seconds from
current time), it is guaranteed that clients will not be able to
retrieve this item after the expiration time arrives (measured by
server time).

Searching finds many links to wiki pages that now 404 (and sadly aren't in the Google cache):
https://code.google.com/p/memcached/wiki/NewProgramming#Expiration
http://code.google.com/p/memcached/wiki/NewCommands#Standard_Protocol

It's just that I've found inconsistencies in the behaviour with negative expiration times, depending on whether binary mode is used or not (see lericson/pylibmc#202), and it's not clear to me whether this is a client or server issue -- or even if negative values are just no longer supported when using binary mode.

Many thanks :-)

100% CPU usage in lru crawler thread

screen shot 2016-08-09 at 5 46 14 pm
Recently, we updated our codes to use version 1.4.29 and did some tests on the new features like lru_crawler. But we noticed one thread that is always consuming 100% thread. After some investigation, we found that it is the lru_crawler and the root cause for it is as following:

In items.c line 1053:

if (lru_crawler_start(i, 0) > 0) {

The second param in lru_crawler_start 'remaining' here should be set to 1 instead of 0. Since in item_crawler_thread, the stop condition is crawlers[i].remaining == 1. Since the remaining passed from lru_crawler_start is set to 0, there will be a infinite loop in item_crawler_thread.

if (search == NULL || (crawlers[i].remaining && --crawlers[i].remaining < 1)) { if (settings.verbose > 2) fprintf(stderr, "Nothing left to crawl for %d\n", i); crawlers[i].it_flags = 0; crawler_count--; crawler_unlink_q((item *)&crawlers[i]); pthread_mutex_unlock(&lru_locks[i]); pthread_mutex_lock(&lru_crawler_stats_lock); crawlerstats[CLEAR_LRU(i)].end_time = current_time; crawlerstats[CLEAR_LRU(i)].run_complete = true; pthread_mutex_unlock(&lru_crawler_stats_lock); continue; }

Function do_lru_crawler_start will only do remaining -1 times crawl, which should be pointed out in the comments.

Support for Aliases

Basic Description: Support for setting an alias to a key, maintained in a separate list to normal keys, which has the property that the alias will never be evicted before the key it points to. Aliases do not affect any of the normal methods, requiring instead specialized alias methods to interact with.

This solves the problem of the key on retrieval being extremely inconvenient for invalidation. Specifically for cases where cache is local to an entry and namespaces can't be used as an effective strategy for invalidation.


Use Case (pseudo-code)

searched_username = "bob";
cacheKey = "user.entry.username.{searched_username}"
model = mc.get(cacheKey);
if (model == NULL) {
    model = user_model_for_username("bob");
    mc.set(cacheKey, model, time() + Month);
    aliasKey = "user.entry.{model.id}.username"
    mc.setAlias(aliasKey, cacheKey);
    return model;
}
else {
    return model;
}

To invalidate the entry with out the mc.setAlias line we have to know the field used to generate the key (in the example username). This can mean we have to either retrieve the entry or pass the information along to the invalidation routine:

cacheKey = "user.entry.username.{username}"
mc.set(cacheKey, NULL, 0);

This looks simple but causes problems such as a delete by ID routine being required to retrieve the model to be able to invalidate the cache. On the other hand if we use aliases we can do it like this:

aliasKey = "user.entry.{id}.username";
cacheKey = mc.getAlias(aliasKey);
mc.set(cacheKey, NULL, 0);

This achieves the same thing but doesn't require any information other then the ID to invalidate any key associated with the model.

In the example I used a routine that searches by username, but it can be extended to something even more complicated such as for example a routine that searches for multiple simultaneous conditions.

The aliases can happily live as an independent system and only really need one property to work:

  • the alias should never be evicted before the key is evicted

In the example I used two calls, but its probably for the best if its simplified to mc.setWithAlias(aliasKey, cacheKey, model, time() + Month) for the first routine and mc.setUsingAlias(aliasKey, NULL, 0) for the invalidation routine.

I don't think aliases should be searched when not explicitly asked for. This should also avoid any performance hit for anyone who don't want to use aliases.

failed cross-compilation

While I was trying to cross-compile memcached for netbsd, but it failed in the test phase.. please help me! Full output is here...

./configure --prefix=/home/vagrant/rumprun-packages/pkgs --with-libevent=/home/vagrant/rumprun-packages/pkgs --host=x86_64-rumprun-netbsd
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-rumprun-netbsd
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for x86_64-rumprun-netbsd-strip... x86_64-rumprun-netbsd-strip
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for x86_64-rumprun-netbsd-gcc... x86_64-rumprun-netbsd-gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether x86_64-rumprun-netbsd-gcc accepts -g... yes
checking for x86_64-rumprun-netbsd-gcc option to accept ISO C89... none needed
checking whether x86_64-rumprun-netbsd-gcc understands -c and -o together... yes
checking for style of include used by make... GNU
checking dependency style of x86_64-rumprun-netbsd-gcc... gcc3
checking how to run the C preprocessor... x86_64-rumprun-netbsd-gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for icc in use... no
checking for clang in use... no
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking whether __SUNPRO_C is declared... no
checking for x86_64-rumprun-netbsd-gcc option to accept ISO C99... -std=gnu99
checking sasl/sasl.h usability... no
checking sasl/sasl.h presence... no
checking for sasl/sasl.h... no
checking for gcov... /usr/bin/gcov
checking for main in -lgcov... no
checking for library containing clock_gettime... none required
checking for library containing socket... none required
checking for library containing gethostbyname... none required
checking for libevent directory... /home/vagrant/rumprun-packages/pkgs
checking for library containing umem_cache_create... no
checking for library containing gethugepagesizes... no
checking for stdbool.h that conforms to C99... yes
checking for _Bool... yes
checking for inttypes.h... (cached) yes
checking for sasl_callback_ft... no
checking for print macros for integers (C99 section 7.8.1)... yes
checking for an ANSI C-conforming const... yes
checking for socklen_t... yes
checking for endianness... configure: error: in /home/vagrant/rumprun-packages/memcached/build': configure: error: cannot run test program while cross compiling Seeconfig.log' for more details
Makefile:25: recipe for target 'build/Makefile' failed
make: *** [build/Makefile] Error 1

Memcached urgent question.

Memcached urgent question:

i wants to know that when if memcached run-out the memory~ would it possibly conver the old data?

cuz we are fixing an tough case right now so thanks for ur help!

add a new command `keys` like redis to list all memcached keys

In the general case, there is no way to list all the keys that a memcached instance is storing. If i had a warm cache and a cold cache, it would be simple to warm up the cold cache using like keys command. So, why memcached doesn't have like keys command to list all the keys?

No support in bash file for automake-1.15

Automake 1.15 was released early this year. I believe this is now the default version on OSX (after installing XCode command line tools).

I created a fork with a change, but wasn't sure if you wanted to deal with merging a whole repo for such a small change. So I figured I'd open up an issue. Let me know if you want me to create a pull request.

The change is here:

Former: locate_binary automake-1.14 automake-1.13 automake-1.12 automake-1.11 automake-1.10 automake-1.9 automake-1.7
New: locate_binary automake-1.15 automake-1.14 automake-1.13 automake-1.12 automake-1.11 automake-1.10 automake-1.9 automake-1.7

Adding a switch for the memlimit in item_cachedump

It is not really a bug but a feature request:

Could you consider to add a switch [for example -ml] for the variable memlimit in items.c which is used in item_cachedump()?

For the moment it is 2MB and because of this only a limited amount of keys is returned when running stats cachedump.

I don't really think it would negatively affect the program when you display a warning - like you already do when setting a higher value for the -I switch.

Will memcached work reliably for implementing a locking mechanism?

http://stackoverflow.com/questions/37458587/can-memcached-be-used-for-locking

Is it possible that Memcache doesn't get reliably due to some performance reason even if the key is present i.e it is has been set before? Lets say the no: of concurrent users is 24, and the request load is higher for example.

How about set, is it possible that a set None i.e a delete operation can fail without a reason leaving the key intact in Memcache? Will add help me in anyway better?

In other words can I Implement a locking Mechanism using Memcached? Name says 'memcached' and I have read that it might actually remove data if it thinks its too much using LRU algorithm which is bad already but it might impact only if the volume is huge. But if get and set is not reliable like Database then it will not work for any logical operations. For Caching it is fine but it usually doesn't work well if you have any logic assosiated with that.

For example, for locking we have a logic that checks if key is present then deny access to the end user. I am seeing that very rarely both users are able to access the resource and I am not able to understand why this is happening.

Error trying to bind to an IPv6 address

When trying to bind memcached to the IPv6 localhost address, I get the following error:

$ memcached -l '[::1]'
Invalid port number: ":1]"failed to listen on TCP port 11211: Success

If I start memcached without arguments it properly binds using IPv6, so the problem seems to be in the parsing of the -l argument.

Make releases available over HTTPS

Is there a desire to make release downloads available over HTTPS?

Although there are available via the GitHub releases interface, there are no published SHA hashes for these releases. The published SHA-1's for the downloads via memcached.org/downloads don't match due to the way the tarballs are generated (modtimes differ, for starters).

I'd be more than happy to donate to cover the entire cost of TLS certs to make this happen (although there is obviously work to do beyond just buying certs). Alternatively, you could make the hashes available via a separate GitHub repository (ala Redis) in a parseable format.

Items expired by flush_all aren't showed as expired in stats

Demonstration:

stats
...
STAT expired_unfetched 0
STAT curr_items 0
set test 0 0 4
test
STORED
stats
...
STAT expired_unfetched 0
STAT curr_items 1
flush_all
OK
stats
...
STAT expired_unfetched 0
STAT curr_items 1

I'd expect the item to show as "expired_unfetched" in this case. Flush_all works, because if I try now to get the item, then it's not returned.

ConcurrentModificationException Memcached multithreading

I am getting java.util.ConcurrentModificationException when I am using Memcached for setting & getting value from multiple thread concurrently.

java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:894)
at java.util.HashMap$EntryIterator.next(HashMap.java:934)
at java.util.HashMap$EntryIterator.next(HashMap.java:932)
at java.util.HashMap.writeObject(HashMap.java:1098)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:975)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1480)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
at com.danga.MemCached.MemCachedClient.set(MemCachedClient.java:763)
at com.danga.MemCached.MemCachedClient.set(MemCachedClient.java:542)
at com.cacheengine.MemoryCacheUtil.addToMemCacheWithExpiryDate(MemoryCacheUtil.java:79)

Please refer link for more details - http://stackoverflow.com/questions/37905851/concurrentmodificationexception-memcached-set-get-from-multithreading

Could you please advise on this issue

Why slab class max is 64?

I find that MAX_NUMBER_OF_SLAB_CLASSES was changed from 201 to 64 in commit 9bce42f.

/* slab class max is a 6-bit number, -1. */
#define MAX_NUMBER_OF_SLAB_CLASSES (63 + 1)

But why limit MAX_NUMBER_OF_SLAB_CLASSES to 64?

build memcached error

I use ./configure and make to build memcache.But fail:

[root@master memcached-1.4.25]# ./configure
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for icc in use... no
checking for clang in use... no
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking whether __SUNPRO_C is declared... no
checking for gcc option to accept ISO C99... -std=gnu99
checking sasl/sasl.h usability... no
checking sasl/sasl.h presence... no
checking for sasl/sasl.h... no
checking for gcov... /usr/bin/gcov
checking for main in -lgcov... yes
checking for library containing clock_gettime... -lrt
checking for library containing socket... none required
checking for library containing gethostbyname... none required
checking for libevent directory... (system)
checking for library containing umem_cache_create... no
checking for library containing gethugepagesizes... -lhugetlbfs
checking for stdbool.h that conforms to C99... yes
checking for _Bool... yes
checking for inttypes.h... (cached) yes
checking for sasl_callback_ft... no
checking for print macros for integers (C99 section 7.8.1)... yes
checking for an ANSI C-conforming const... yes
checking for socklen_t... yes
checking for endianness... little
checking for htonll... no
checking for library containing pthread_create... none required
checking for mlockall... yes
checking for getpagesizes... yes
checking for memcntl... no
checking for sigignore... yes
checking for clock_gettime... yes
checking for accept4... yes
checking for alignment... none
checking for GCC atomics... yes
checking for setppriv... no
checking umem.h usability... no
checking umem.h presence... no
checking for umem.h... no
checking for xml2rfc... /usr/bin/xml2rfc
checking for xsltproc... /usr/bin/xsltproc
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating doc/Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands

[root@master memcached-1.4.25]# make
make all-recursive
make[1]: Entering directory /root/memcached-1.4.25/memcached-1.4.25' Making all in doc make[2]: Entering directory/root/memcached-1.4.25/memcached-1.4.25/doc'
make[2]: *** No rule to make target .full', needed byprotocol-binary.txt'. Stop.
make[2]: Leaving directory /root/memcached-1.4.25/memcached-1.4.25/doc' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory/root/memcached-1.4.25/memcached-1.4.25'
make: *** [all] Error 2

Test t/refhang.t failing on s390x

The testsuite fail for the s390x on the buildd Debian system:

[...]
t/noreply.t ................. ok

#   Failed test 'some ooms happened'
#   at t/refhang.t line 51.

#   Failed test 'nonzero lrutail_reflocked'
#   at t/refhang.t line 53.
#          got: '0'
#     expected: anything else

#   Failed test 'nonzero total lrutail_reflocked'
#   at t/refhang.t line 56.
#          got: '0'
#     expected: anything else
# Looks like you failed 3 tests of 127.
t/refhang.t ................. 
Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/127 subtests 
t/slabs-reassign-chunked.t .. ok
[...]

I'll try to give an access to such box to try to investigate this issue, but i can't promise...
I've opened a Debian bug to follow-up this issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=835456

Source: https://buildd.debian.org/status/fetch.php?pkg=memcached&arch=s390x&ver=1.4.31-1&stamp=1471823136

Document the supported SASL mechanisms (eg PLAIN, CRAM-MD5)

There are a number of defined SASL mechanisms:
https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer#SASL_mechanisms

I tried looking on the following wiki pages, to see which memcache's binary protocol supported, with no luck:
https://github.com/memcached/memcached/wiki/SASLAuthProtocol
https://github.com/memcached/memcached/wiki/SASLHowto

The following test would imply that the only supported mechanisms are PLAIN and CRAM-MD5:
https://github.com/memcached/memcached/blob/1.4.29/t/binary-sasl.t#L146

  1. Is this correct?
  2. Please can this be documented on one (or both) of those wiki pages?
  3. Is there any plan to support a TLS based approach? (Given PLAIN transfers credentials in plaintext, CRAM-MD5 can be bruteforced, and neither protect against post-authentication modification of payload.)

Many thanks :-)

memcached statistics total_items is less than cmd_set

The statistics on my memcached server show a strange relationship: total_items is less than cmd_set. I am not entirely sure that this is a bug, but it looks quite strange to me. I checked the statistics description in doc/protocol.txt and asked on http://stackoverflow.com/questions/37324108 without gaining further insight.

The only operations run on this server are "set" and "get", nothing else like "add", "replace", "delete" or CAS based operations.

When I go through the memcached source code, I see that any "normal" set (w/o CAS) will either replace or write the item, and both will increment total_items.

do_store_item() {
    // ...
    if (old_it != NULL)
        item_replace(old_it, it, hv);
    else
        do_item_link(it, hv);
    // ...
}

do_item_link() increases total_items, and item_replace() also calls do_item_link(). Then how can total_items be less than cmd_set?

Excerpt from memcached stats (numbers indented for readability):

STAT cmd_set          12827359728
STAT total_items       4237422053
STAT curr_items          60745375
STAT expired_unfetched 9898430934
STAT evicted_unfetched   30415090
STAT evictions           30421532
STAT reclaimed         9900995350

t/slabs-reassign2.t is flaky

the "some rescues happened" and "some evictions happened" checks are racing the slab rebalancer actually moving the pages around.

Need to restructure to specifically test those conditions by either pausing the balancer or checking in the other rebalance test which explicitly moves pages around.

build failure with musl libc

Building with musl libc fails with this error:

gcc -DHAVE_CONFIG_H -I.  -DNDEBUG   -g -O2 -pthread -pthread -Wall -Werror -pedantic -Wmissing-prototypes -Wmissing-declarations -Wredundant-decls -MT memcached-slabs.o -MD -MP -MF .deps/memcached-slabs.Tpo -c -o memcached-slabs.o `test -f 'slabs.c' || echo './'`slabs.c
In file included from slabs.c:13:0:
/usr/include/sys/signal.h:1:2: error: #warning redirecting incorrect #include <sys/signal.h> to <signal.h> [-Werror=cpp]
 #warning redirecting incorrect #include <sys/signal.h> to <signal.h>
  ^
cc1: all warnings being treated as errors

The warning is correct according posix.

comm didn't initialize, is this a bug?

file:memcached.c
version:1.4.25
line:3433
function:process_command

static void process_command(conn *c, char *command) {

token_t tokens[MAX_TOKENS];
size_t ntokens;
int comm;

assert(c != NULL);

MEMCACHED_PROCESS_COMMAND_START(c->sfd, c->rcurr, c->rbytes);

if (settings.verbose > 1)
    fprintf(stderr, "<%d %s\n", c->sfd, command);

/*
 * for commands set/add/replace, we build an item and read the data
 * directly into it, then continue in nread_complete().
 */

c->msgcurr = 0;
c->msgused = 0;
c->iovused = 0;
if (add_msghdr(c) != 0) {
    out_of_memory(c, "SERVER_ERROR out of memory preparing response");
    return;
}

ntokens = tokenize_command(command, tokens, MAX_TOKENS);
if (ntokens >= 3 &&
    ((strcmp(tokens[COMMAND_TOKEN].value, "get") == 0) ||
     (strcmp(tokens[COMMAND_TOKEN].value, "bget") == 0))) {

    process_get_command(c, tokens, ntokens, false);

} else if ((ntokens == 6 || ntokens == 7) &&
           ((strcmp(tokens[COMMAND_TOKEN].value, "add") == 0 && (comm = NREAD_ADD)) ||
            (strcmp(tokens[COMMAND_TOKEN].value, "set") == 0 && (comm = NREAD_SET)) ||
            (strcmp(tokens[COMMAND_TOKEN].value, "replace") == 0 && (comm = NREAD_REPLACE)) ||
            (strcmp(tokens[COMMAND_TOKEN].value, "prepend") == 0 && (comm = NREAD_PREPEND)) ||
            (strcmp(tokens[COMMAND_TOKEN].value, "append") == 0 && (comm = NREAD_APPEND)) )) {

    process_update_command(c, tokens, ntokens, comm, false);

} else if ((ntokens == 7 || ntokens == 8) && (strcmp(tokens[COMMAND_TOKEN].value, "cas") == 0 && (comm = NREAD_CAS))) {

    process_update_command(c, tokens, ntokens, comm, true);

} else if ((ntokens == 4 || ntokens == 5) && (strcmp(tokens[COMMAND_TOKEN].value, "incr") == 0)) {

    process_arithmetic_command(c, tokens, ntokens, 1);

} else if (ntokens >= 3 && (strcmp(tokens[COMMAND_TOKEN].value, "gets") == 0)) {

    process_get_command(c, tokens, ntokens, true);

} else if ((ntokens == 4 || ntokens == 5) && (strcmp(tokens[COMMAND_TOKEN].value, "decr") == 0)) {

    process_arithmetic_command(c, tokens, ntokens, 0);

} else if (ntokens >= 3 && ntokens <= 5 && (strcmp(tokens[COMMAND_TOKEN].value, "delete") == 0)) {

    process_delete_command(c, tokens, ntokens);

} else if ((ntokens == 4 || ntokens == 5) && (strcmp(tokens[COMMAND_TOKEN].value, "touch") == 0)) {

    process_touch_command(c, tokens, ntokens);

} else if (ntokens >= 2 && (strcmp(tokens[COMMAND_TOKEN].value, "stats") == 0)) {

    process_stat(c, tokens, ntokens);

} else if (ntokens >= 2 && ntokens <= 4 && (strcmp(tokens[COMMAND_TOKEN].value, "flush_all") == 0)) {
    time_t exptime = 0;
    rel_time_t new_oldest = 0;

    set_noreply_maybe(c, tokens, ntokens);

    pthread_mutex_lock(&c->thread->stats.mutex);
    c->thread->stats.flush_cmds++;
    pthread_mutex_unlock(&c->thread->stats.mutex);

    if (!settings.flush_enabled) {
        // flush_all is not allowed but we log it on stats
        out_string(c, "CLIENT_ERROR flush_all not allowed");
        return;
    }

    if (ntokens != (c->noreply ? 3 : 2)) {
        exptime = strtol(tokens[1].value, NULL, 10);
        if(errno == ERANGE) {
            out_string(c, "CLIENT_ERROR bad command line format");
            return;
        }
    }

    /*
      If exptime is zero realtime() would return zero too, and
      realtime(exptime) - 1 would overflow to the max unsigned
      value.  So we process exptime == 0 the same way we do when
      no delay is given at all.
    */
    if (exptime > 0) {
        new_oldest = realtime(exptime);
    } else { /* exptime == 0 */
        new_oldest = current_time;
    }

    if (settings.use_cas) {
        settings.oldest_live = new_oldest - 1;
        if (settings.oldest_live <= current_time)
            settings.oldest_cas = get_cas_id();
    } else {
        settings.oldest_live = new_oldest;
    }
    out_string(c, "OK");
    return;

} else if (ntokens == 2 && (strcmp(tokens[COMMAND_TOKEN].value, "version") == 0)) {

    out_string(c, "VERSION " VERSION);

} else if (ntokens == 2 && (strcmp(tokens[COMMAND_TOKEN].value, "quit") == 0)) {

    conn_set_state(c, conn_closing);

} else if (ntokens == 2 && (strcmp(tokens[COMMAND_TOKEN].value, "shutdown") == 0)) {

    if (settings.shutdown_command) {
        conn_set_state(c, conn_closing);
        raise(SIGINT);
    } else {
        out_string(c, "ERROR: shutdown not enabled");
    }

} else if (ntokens > 1 && strcmp(tokens[COMMAND_TOKEN].value, "slabs") == 0) {
    if (ntokens == 5 && strcmp(tokens[COMMAND_TOKEN + 1].value, "reassign") == 0) {
        int src, dst, rv;

        if (settings.slab_reassign == false) {
            out_string(c, "CLIENT_ERROR slab reassignment disabled");
            return;
        }

        src = strtol(tokens[2].value, NULL, 10);
        dst = strtol(tokens[3].value, NULL, 10);

        if (errno == ERANGE) {
            out_string(c, "CLIENT_ERROR bad command line format");
            return;
        }

        rv = slabs_reassign(src, dst);
        switch (rv) {
        case REASSIGN_OK:
            out_string(c, "OK");
            break;
        case REASSIGN_RUNNING:
            out_string(c, "BUSY currently processing reassign request");
            break;
        case REASSIGN_BADCLASS:
            out_string(c, "BADCLASS invalid src or dst class id");
            break;
        case REASSIGN_NOSPARE:
            out_string(c, "NOSPARE source class has no spare pages");
            break;
        case REASSIGN_SRC_DST_SAME:
            out_string(c, "SAME src and dst class are identical");
            break;
        }
        return;
    } else if (ntokens == 4 &&
        (strcmp(tokens[COMMAND_TOKEN + 1].value, "automove") == 0)) {
        process_slabs_automove_command(c, tokens, ntokens);
    } else {
        out_string(c, "ERROR");
    }
} else if (ntokens > 1 && strcmp(tokens[COMMAND_TOKEN].value, "lru_crawler") == 0) {
    if (ntokens == 4 && strcmp(tokens[COMMAND_TOKEN + 1].value, "crawl") == 0) {
        int rv;
        if (settings.lru_crawler == false) {
            out_string(c, "CLIENT_ERROR lru crawler disabled");
            return;
        }

        rv = lru_crawler_crawl(tokens[2].value);
        switch(rv) {
        case CRAWLER_OK:
            out_string(c, "OK");
            break;
        case CRAWLER_RUNNING:
            out_string(c, "BUSY currently processing crawler request");
            break;
        case CRAWLER_BADCLASS:
            out_string(c, "BADCLASS invalid class id");
            break;
        case CRAWLER_NOTSTARTED:
            out_string(c, "NOTSTARTED no items to crawl");
            break;
        }
        return;
    } else if (ntokens == 4 && strcmp(tokens[COMMAND_TOKEN + 1].value, "tocrawl") == 0) {
        uint32_t tocrawl;
         if (!safe_strtoul(tokens[2].value, &tocrawl)) {
            out_string(c, "CLIENT_ERROR bad command line format");
            return;
        }
        settings.lru_crawler_tocrawl = tocrawl;
        out_string(c, "OK");
        return;
    } else if (ntokens == 4 && strcmp(tokens[COMMAND_TOKEN + 1].value, "sleep") == 0) {
        uint32_t tosleep;
        if (!safe_strtoul(tokens[2].value, &tosleep)) {
            out_string(c, "CLIENT_ERROR bad command line format");
            return;
        }
        if (tosleep > 1000000) {
            out_string(c, "CLIENT_ERROR sleep must be one second or less");
            return;
        }
        settings.lru_crawler_sleep = tosleep;
        out_string(c, "OK");
        return;
    } else if (ntokens == 3) {
        if ((strcmp(tokens[COMMAND_TOKEN + 1].value, "enable") == 0)) {
            if (start_item_crawler_thread() == 0) {
                out_string(c, "OK");
            } else {
                out_string(c, "ERROR failed to start lru crawler thread");
            }
        } else if ((strcmp(tokens[COMMAND_TOKEN + 1].value, "disable") == 0)) {
            if (stop_item_crawler_thread() == 0) {
                out_string(c, "OK");
            } else {
                out_string(c, "ERROR failed to stop lru crawler thread");
            }
        } else {
            out_string(c, "ERROR");
        }
        return;
    } else {
        out_string(c, "ERROR");
    }
} else if ((ntokens == 3 || ntokens == 4) && (strcmp(tokens[COMMAND_TOKEN].value, "verbosity") == 0)) {
    process_verbosity_command(c, tokens, ntokens);
} else {
    out_string(c, "ERROR");
}
return;

}

everybody needs support for buckets or regions :-)

This is in fact a feature request ;-)

or it's me and it is already there?

--EDIT--
Just to be sure, i know about multiple instances, but if your not able to get them we need a native way to simulate this within one - lets say shared - instance

Possible issue with slab rebalance.

I have been looking over the code in slabs.h/slabs.c and trying to understand how the source and destination slabs are chosen for the rebalancing algorithm.

In slab_rebalance_finish I see that src and destination are set to zero, presumably for the next run.

809 slab_rebal.s_clsid = 0;
810 slab_rebal.d_clsid = 0;

These vars are then used to index into the slabs array in slab_rebalance_alloc, slab_rebalance_move, and slab_rebalance_finish routines, however, I do not see anywhere in which s_clsid or d_clsid are ever set to anything besides 0 (aside from the slab_reassign command). In other words it appears that the source and destination are always set to zero and the conditional on line 508 of slabs.c will always result in no_go = -2.

It does not appear that the values are modified outside of slabs.c either.

if (slab_rebal.s_clsid < POWER_SMALLEST || slab_rebal.s_clsid > power_largest || slab_rebal.d_clsid < SLAB_GLOBAL_PAGE_POOL || slab_rebal.d_clsid > power_largest || slab_rebal.s_clsid == slab_rebal.d_clsid) no_go = -2;

memcached.org download invalid gzip file

official site memcached.org version of download at http://www.memcached.org/files/memcached-1.4.26.tar.gz is not 100% valid see below screenshot the one at 1600KB is memcached.org downloaded one and below is one i redid a tar + gzip on

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Error: memcached-1.4.26.tar.gz extraction failed.

image

problem is when i download from windows 8 as well as centos 6.8 and 7.2

Life cycle management of Data

Dear Team,

I am looking for cache data that will be persisted on timely bases.

Can you please let me know what is the time period for refresh of data/object or what algorithm is fallowed to send it to GC/Destroy.

Thanks

Out of memory issue with lru_maintainer on

I tried to turned on slab_automove=1 in version 1.4.29. But it seems that the automove will only be executed when lru_maintainer is on. I checked the code and the logic has been move to lru_maintainer thread. But I did not see (maybe I missed) that it is mentioned in the protocol. So I turned on the lru_maintainer but I found tons of out of memory error when the memory is fully used. It seems that the lru tail eviction is not happening when there is not free chunks available. I did not see the issue when lru_maintainer is off. Could anyone help me take a look?

Feature Request

  1. Update Memcached to work with PHP 7. Currently only working until PHP 5.6
  2. For the Labels you have created add Awaiting Reply so that users can know that you are waiting for them to respond

memcached error 21

I am using in my Debian 8.2:
memcached 1.4.21-1.1
libmemcached11:amd64 1.0.18-4
pylibmc==1.5.0

and got this errors:

'shallow': False,
'url_rule': <Rule '/' (HEAD, POST, OPTIONS, GET) -> index_page.index>,
'view_args': {}}
Traceback (most recent call last):
File "/srv/venv/project/local/lib/python2.7/site-packages/flask_cache/init.py", line 540, in decorated_function
timeout=decorated_function.cache_timeout)
File "/srv/venv/project/local/lib/python2.7/site-packages/werkzeug/contrib/cache.py", line 399, in set
return self._client.set(key, value, timeout)
Error: error 21 from memcached_set: SUCCESS

what it is meen and how i can fix it?

Test suite stuck at t/watcher.t

As talk with @dormando , test suite is tuck on t/watcher.t when running make test on a Debian sid amd64 chroot, patching t/watcher.t with:

Index: memcached/t/watcher.t                                                                                                                                                                                 
===================================================================              
--- memcached.orig/t/watcher.t                                                   
+++ memcached/t/watcher.t                                                        
@@ -33,9 +33,10 @@ for (1 .. 80000) {                                            
 }                                                                               

 # Let the logger thread catch up before we start reading.                       
-sleep 1;                                                                        
+sleep 10;                                                                       
 #print STDERR "RESULT: $res\n";                                                 
 while (my $log = <$watcher>) {                                                  
+   print STDERR $log;                                                           
     next unless $log =~ m/skipped/;                                             
     like($log, qr/skipped=/, "skipped some lines");                             
     # This should unjam more of the text.

reveal:

prove -v t/watcher.t
profiling:/home/gui/dev/debian/packages/memcached/memcached_debug-memcached.gcda:Merge mismatch for function 77
ts=1467143587.562316 gid=1 type=item_get key=foo status=not_found
ts=1467143587.562953 gid=2 type=item_get key=foo1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX status=not_found
[...]
ts=1467143588.57216 gid=9974 type=item_get key=foo9973XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX status=not_found
ts=1467143588.57253 gid=9975 type=item_get key=foo9974XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX status=not_found
ts=1467143588.57290 gid=9976 type=item_get key=foo9975XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX status=not_found
Timeout.. killing the process

#   Failed test 'saw a real log line after a skip'
#   at t/watcher.t line 48.
#                   undef
#     doesn't match '(?^:ts=\d+\.\d+\ gid=\d+ type=item_get)'
t/watcher.t ..
1..8
ok 1 - watcher enabled
ok 2 - basic get works
not ok 3 - saw a real log line after a skip
Failed 6/8 subtests

Test Summary Report
-------------------
t/watcher.t (Wstat: 13 Tests: 3 Failed: 1)
  Failed test:  3
  Non-zero wait status: 13
  Parse errors: Bad plan.  You planned 8 tests but ran 3.
Files=1, Tests=3, 600 wallclock secs ( 0.03 usr  0.00 sys +  0.22 cusr  2.27 csys =  2.52 CPU)
Result: FAIL

Reproduced on a rpi2 :)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.