gae-fastcounters is a library providing fast counters for the Python runtime on Google App Engine. It is extremely fast, lightweight (one file), and easy to use.
- Lightweight: One short file.
- Fast and Efficient
- Order of magnitude faster than sharding-based counter implementations
- Uses memcache to minimize datastore interaction and update times
- Uses the task queue to defer persistent updates until after the user's request is handled
- Frequency of datastore writes is minimized by only writing at most once per user-defined update interval
- Can fetch multiple counters en-masse
- Simple to Use
- Just three simple methods:
incr
,get_count
, andget_counts
- Just three simple methods:
- Counter values are limited to the range
[-2**63, 2**63 - 1]
- If the database update interval is not set to be immediate, then counters may undercount if an unpersisted portion of the count is evicted from memcache before it is persisted to the datastore. By default, the update interval is 10 seconds.
After downloading and unpacking gae-fastcounters, copy the 'fastcounter.py' file into your app's root directory.
By default, changes will only be persisted to the datastore if a change to the
counter being updated hasn't been persisted in the last 10 seconds. You can
modify this interval to suit your needs by passing an update_interval
parameter to fastcounter.incr()
.
If you pass a very small update_interval
, then all or nearly all changes will
trigger a task to be immediately enqueued to persist the change to the
datastore. The smaller the interval, the smaller the chance that the counter
will lose a change. However, updating them more frequently will use more
resources since every update will ultimately trigger a transactional database
write. Also, beware contention - App Engine can only support so many updates
per second to any given entity, and fewer for transactional updates (it has
improved over time, but more than once per second is still pushing it; YMMY).
Modifying Counters
import fastcounter
# increment by one
fastcounter.incr('my_counter0')
# decrement by one
fastcounter.incr('my_counter1', delta=-1)
# persist this one more frequently
fastcounter.incr('my_counter2', update_interval=5)
Retrieving Counters
import fastcounter
value0 = fastcounter.get_count('my_counter0')
# if you ask for a counter which has never been used before, its value is 0
value9 = fastcounter.get_count('my_counter9')
# getting counters in bulk is far more efficient (every counter retrieval
# call requires a round-trip to the datastore) values =
values = fastcounter.get_counts(['my_counter%d' % i for i in xrange(100)])
Author: Pocket Gems
Updated: 2012-Feb-06 (v0.1)
License: Apache License Version 2.0
If you discover a problem, please report it on the gae-fastcounters issues page.