scrapy 2.3 在蜘蛛中提取數(shù)據(jù)

讓我們回到蜘蛛身邊。到目前為止，它還沒有提取任何數(shù)據(jù)，特別是將整個(gè)HTML頁(yè)面保存到一個(gè)本地文件中。讓我們把上面的提取邏輯集成到蜘蛛中。

剪貼蜘蛛通常會(huì)生成許多字典，其中包含從頁(yè)面中提取的數(shù)據(jù)。為此，我們使用 ?yield? 回調(diào)中的python關(guān)鍵字，如下所示：

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
        'http://quotes.toscrape.com/page/2/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

如果運(yùn)行這個(gè)spider，它將用日志輸出提取的數(shù)據(jù)：

2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/>
{'tags': ['life', 'love'], 'author': 'André Gide', 'text': '“It is better to be hated for what you are than to be loved for what you are not.”'}
2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/>
{'tags': ['edison', 'failure', 'inspirational', 'paraphrased'], 'author': 'Thomas A. Edison', 'text': "“I have not failed. I've just found 10,000 ways that won't work.”"}