scrapy 2.3 Shell會(huì)話示例

下面是一個(gè)典型的shell會(huì)話的示例，我們從https://scray.org頁(yè)面，然后繼續(xù)刮https://old.reddit.com/第頁(yè)。最后，我們修改（Reddit）請(qǐng)求方法，以發(fā)布并重新獲取它，得到一個(gè)錯(cuò)誤。我們通過(guò)鍵入Ctrl-D（在Unix系統(tǒng)中）或在Windows中鍵入Ctrl-Z來(lái)結(jié)束會(huì)話。

請(qǐng)記住，在這里提取的數(shù)據(jù)在您嘗試時(shí)可能不相同，因?yàn)檫@些頁(yè)面不是靜態(tài)的，在您測(cè)試時(shí)可能已經(jīng)更改了。這個(gè)例子的唯一目的是讓您熟悉下腳料外殼的工作原理。

首先，我們發(fā)射炮彈：

scrapy shell 'https://scrapy.org' --nolog

注解

當(dāng)從命令行運(yùn)行Scrapy shell時(shí)，記住總是用引號(hào)括住url，否則url包含參數(shù)（即 ?&? 字符）不起作用。

在Windows上，使用雙引號(hào)：

scrapy shell "https://scrapy.org" --nolog

然后，shell獲取URL（使用scrapy下載器）并打印可用對(duì)象和有用快捷方式的列表（您會(huì)注意到這些行都以 ?[s]? 前綴）：

[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   crawler    <scrapy.crawler.Crawler object at 0x7f07395dd690>
[s]   item       {}
[s]   request    <GET https://scrapy.org>
[s]   response   <200 https://scrapy.org/>
[s]   settings   <scrapy.settings.Settings object at 0x7f07395dd710>
[s]   spider     <DefaultSpider 'default' at 0x7f0735891690>
[s] Useful shortcuts:
[s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s]   fetch(req)                  Fetch a scrapy.Request and update local objects
[s]   shelp()           Shell help (print this help)
[s]   view(response)    View response in a browser

>>>

之后，我們可以開(kāi)始玩對(duì)象：

>>> response.xpath('//title/text()').get()
'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework'

>>> fetch("https://old.reddit.com/")

>>> response.xpath('//title/text()').get()
'reddit: the front page of the internet'

>>> request = request.replace(method="POST")

>>> fetch(request)

>>> response.status
404

>>> from pprint import pprint

>>> pprint(response.headers)
{'Accept-Ranges': ['bytes'],
 'Cache-Control': ['max-age=0, must-revalidate'],
 'Content-Type': ['text/html; charset=UTF-8'],
 'Date': ['Thu, 08 Dec 2016 16:21:19 GMT'],
 'Server': ['snooserv'],
 'Set-Cookie': ['loid=KqNLou0V9SKMX4qb4n; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
                'loidcreated=2016-12-08T16%3A21%3A19.445Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
                'loid=vi0ZVe4NkxNWdlH7r7; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
                'loidcreated=2016-12-08T16%3A21%3A19.459Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure'],
 'Vary': ['accept-encoding'],
 'Via': ['1.1 varnish'],
 'X-Cache': ['MISS'],
 'X-Cache-Hits': ['0'],
 'X-Content-Type-Options': ['nosniff'],
 'X-Frame-Options': ['SAMEORIGIN'],
 'X-Moose': ['majestic'],
 'X-Served-By': ['cache-cdg8730-CDG'],
 'X-Timer': ['S1481214079.394283,VS0,VE159'],
 'X-Ua-Compatible': ['IE=edge'],
 'X-Xss-Protection': ['1; mode=block']}