liuxingming / scrapy_redis_bloomfilter Goto Github PK

dupefilter.py 文件里面from scrapy.dupefilters import BaseDupeFilter 应该改成 from scrapy.dupefilter import BaseDupeFilter

比如网站 http://www.xxx.com/list-1 表示第1页，我需要爬取第1页里面特定子 url 的新闻。比如第一天使用 scrapy_redis_bloomfilter 爬取了 http://www.xxx.com/list-1 的新闻，然后第二天由于网站更新了 http://www.xxx.com/list-1 显示子 url 新闻就不一样了。这时候问题就来了，需求是增量爬取子 url 新闻，但是scrapy_redis_bloomfilter 的去重就会导致 http://www.xxx.com/list-1 根本就不再爬取了，这时候就爬不到最新的新闻了，这个问题怎么解决呢？麻烦大神回复下呢

In file 'Scrapy_Redis_Bloomfilter/scrapyWithBloomfilter_demo/scrapyWithBloomfilter_demo/scrapy_redis/BloomfilterOnRedis.py', code lines of 33-42:

def isContains(self, str_input):
    if not str_input:
        return False
    ret = True

    name = self.key + str(int(str_input[0:2], 16) % self.blockNum)
    for f in self.hashfunc:
        loc = f.hash(str_input)
        ret = ret & self.server.getbit(name, loc)
    return ret

When getbit returns 0, it could tell the non-contains. So I think it'd be better to add 2 more code lines like this to speed up this function:

def isContains(self, str_input):
    if not str_input:
        return False
    ret = True

    name = self.key + str(int(str_input[0:2], 16) % self.blockNum)
    for f in self.hashfunc:
        loc = f.hash(str_input)
        ret = ret & self.server.getbit(name, loc)
        if not ret:
            break
    return ret

or might be even better without the '&' operation:

def isContains(self, str_input):
    if not str_input:
        return False
    ret = True

    name = self.key + str(int(str_input[0:2], 16) % self.blockNum)
    for f in self.hashfunc:
        loc = f.hash(str_input)
        ret = self.server.getbit(name, loc)
        if not ret:
            break
    return ret

TypeError: zadd() keywords must be strings
有这样的错误

--- ---
File "/usr/local/lib/python3.6/dist-packages/twisted/internet/base.py", line 878, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/usr/local/lib/python3.6/dist-packages/scrapy/utils/reactor.py", line 41, in call
return self._func(*self._a, **self._kw)
File "/usr/local/lib/python3.6/dist-packages/scrapy/core/engine.py", line 135, in _next_request
self.crawl(request, spider)
File "/usr/local/lib/python3.6/dist-packages/scrapy/core/engine.py", line 210, in crawl
self.schedule(request, spider)
File "/usr/local/lib/python3.6/dist-packages/scrapy/core/engine.py", line 216, in schedule
if not self.slot.scheduler.enqueue_request(request):
File "/root/spiders/yunqiCrawl/yunqiCrawl/scrapy_redis/scheduler.py", line 82, in enqueue_request
self.queue.push(request)
File "/root/spiders/yunqiCrawl/yunqiCrawl/scrapy_redis/queue.py", line 84, in push
self.server.zadd(self.key, **pairs)
builtins.TypeError: zadd() keywords must be strings