0
点赞
收藏
分享

微信扫一扫

scrapy 使用自定义的过滤器

SPEIKE 2022-08-01 阅读 77


from scrapy_redis.dupefilter import RFPDupeFilter



class CustomFilter(RFPDupeFilter):
def request_seen(self, request):
"""Returns True if request was already seen.

Parameters
----------
request : scrapy.http.Request

Returns
-------
bool

"""
if in request.url:
return False
fp = self.request_fingerprint(request)
# This returns the number of values added, zero if already exists.
added = self.server.sadd(self.key, fp)
return added == 0

这边我写了一个自定义的过滤器,继承于scrapy-redis中的。因为我有个需求是,这条url不过滤。

settings.py

DUPEFILTER_CLASS = 'tutorial.CustomFilter.CustomFilter'

注意我项目名字是tutorial


举报

相关推荐

0 条评论