scrapy 2.3 重復(fù)篩選器

查找重復(fù)項(xiàng)并刪除已處理的項(xiàng)的篩選器。假設(shè)我們的項(xiàng)目有一個(gè)唯一的ID，但是我們的spider返回具有相同ID的多個(gè)項(xiàng)目：

from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem

class DuplicatesPipeline:

    def __init__(self):
        self.ids_seen = set()

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        if adapter['id'] in self.ids_seen:
            raise DropItem(f"Duplicate item found: {item!r}")
        else:
            self.ids_seen.add(adapter['id'])
            return item