Giter VIP home page Giter VIP logo

pyrebloom-ng's Introduction

pyreBloom-ng

Python library which implements a Redis-backed Bloom filter.
This is a fork of pyreBloom, but a bit faster, has a better API and supports Python 2.6+, 3.3+, PyPy 2 and PyPy 3.
This repo fork from <https://github.com/leovp/pyreBloom-ng> but change a litte for use and guide|

Installation

pyreBloom-ng requires hiredis library, Cython and a C compiler.

Install hiredis:

# On Mac:
brew install hiredis

# On Debian:
apt-get install libhiredis-dev

# From source:
git clone https://github.com/redis/hiredis
cd hiredis && make && sudo make install

Install the latest stable library version:

pip install Cython
pip install git+https://github.com/C1tas/pyreBloom-ng

Instantiate a pyreBloom filter, giving it a redis key prefix, a capacity, and an error rate:

from pyreBloom import PyreBloom

# Important: ALL keys are bytes and NOT unicode strings.
# Redis doesn't care about unicode at all.
f = PyreBloom(b'key_prefix', 10000, 0.01)

# You can find out how many bits this will theoretically consume
f.bits

# And how many hashes are needed to satisfy the false positive rate
f.hashes

Test

f.add(b'http://asd.com')
# True
b'http://asd.com' in f
# True
b'http://asd.com/1' in f
# False

Easily add data to a filter using a set-like interface:

# Add one value at a time (slow).
f.add(b'bytestuff')

# Or use batch operations (faster).
data = [os.urandom(8) for _ in range(1024)]
f.update(data)
# Alternative: f += data

Now you can perform membership tests:

# Test one value at a time (slow).
>>> obj = b'\x00\x01\x02'
>>> obj in f
True

# Use batch operations (faster).
# Note: pyreBloom.intersection() returns a list of values
# which are found in a Bloom filter. It makes sense when
# you consider it a set-like operation.
f.update([b'0', b'1', b'2', b'3', b'4'])
found = f.intersection([b'3', b'4', b'5', b'6'])
# Alternative: found = f & [b'3', b'4', b'5', b'6']
# found is now [b'3', b'4']

License

Both pyreBloom and pyreBloom-ng are distributed under the terms of the MIT license.

See the bundled LICENSE file for more details.

pyrebloom-ng's People

Contributors

c1tas avatar leovp avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.