閱讀(2.1k) 書(shū)簽贊(0) 我要糾錯(cuò)

Beautiful Soup 4 遍歷

2021-05-21 15:26 更新

還拿"愛(ài)麗絲夢(mèng)游仙境"的文檔來(lái)做例子:

html_doc = """
<html><head><title>The Dormouse's story</title></head>
    <body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  class="sister" id="link1">Elsie</a>,
<a  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  class="sister" id="link2">Lacie</a> and
<a  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, 'html.parser')

通過(guò)這段例子來(lái)演示怎樣從文檔的一段內(nèi)容找到另一段內(nèi)容

子節(jié)點(diǎn)

一個(gè)Tag可能包含多個(gè)字符串或其它的Tag,這些都是這個(gè)Tag的子節(jié)點(diǎn).Beautiful Soup提供了許多操作和遍歷子節(jié)點(diǎn)的屬性.

注意: Beautiful Soup中字符串節(jié)點(diǎn)不支持這些屬性,因?yàn)樽址疀](méi)有子節(jié)點(diǎn)

tag的名字

操作文檔樹(shù)最簡(jiǎn)單的方法就是告訴它你想獲取的tag的name.如果想獲取 <head> 標(biāo)簽,只要用 ?soup.head? :

soup.head
# <head><title>The Dormouse's story</title></head>

soup.title
# <title>The Dormouse's story</title>

這是個(gè)獲取tag的小竅門(mén),可以在文檔樹(shù)的tag中多次調(diào)用這個(gè)方法.下面的代碼可以獲取<body>標(biāo)簽中的第一個(gè)<b>標(biāo)簽:

soup.body.b
# <b>The Dormouse's story</b>

通過(guò)點(diǎn)取屬性的方式只能獲得當(dāng)前名字的第一個(gè)tag:

soup.a
# <a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link1">Elsie</a>

如果想要得到所有的<a>標(biāo)簽,或是通過(guò)名字得到比一個(gè)tag更多的內(nèi)容的時(shí)候,就需要用到 Searching the tree 中描述的方法,比如: find_all()

soup.find_all('a')
# [<a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link1">Elsie</a>,
#  <a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link2">Lacie</a>,
#  <a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link3">Tillie</a>]

.contents 和 .children

tag的 ?.contents? 屬性可以將tag的子節(jié)點(diǎn)以列表的方式輸出:

head_tag = soup.head
head_tag
# <head><title>The Dormouse's story</title></head>

head_tag.contents
[<title>The Dormouse's story</title>]

title_tag = head_tag.contents[0]
title_tag
# <title>The Dormouse's story</title>
title_tag.contents
# [u'The Dormouse's story']

?BeautifulSoup? 對(duì)象本身一定會(huì)包含子節(jié)點(diǎn),也就是說(shuō)<html>標(biāo)簽也是 ?BeautifulSoup? 對(duì)象的子節(jié)點(diǎn):

len(soup.contents)
# 1
soup.contents[0].name
# u'html'

字符串沒(méi)有 ?.contents? 屬性,因?yàn)樽址疀](méi)有子節(jié)點(diǎn):

text = title_tag.contents[0]
text.contents
# AttributeError: 'NavigableString' object has no attribute 'contents'

通過(guò)tag的 ?.children? 生成器,可以對(duì)tag的子節(jié)點(diǎn)進(jìn)行循環(huán):

for child in title_tag.children:
    print(child)
    # The Dormouse's story

.descendants

?.contents? 和 ?.children? 屬性僅包含tag的直接子節(jié)點(diǎn).例如,<head>標(biāo)簽只有一個(gè)直接子節(jié)點(diǎn)<title>

head_tag.contents
# [<title>The Dormouse's story</title>]

但是<title>標(biāo)簽也包含一個(gè)子節(jié)點(diǎn):字符串 “The Dormouse’s story”,這種情況下字符串 “The Dormouse’s story”也屬于<head>標(biāo)簽的子孫節(jié)點(diǎn)。?.descendants? 屬性可以對(duì)所有tag的子孫節(jié)點(diǎn)進(jìn)行遞歸循環(huán) [5] :

for child in head_tag.descendants:
    print(child)
    # <title>The Dormouse's story</title>
    # The Dormouse's story

上面的例子中, <head>標(biāo)簽只有一個(gè)子節(jié)點(diǎn),但是有2個(gè)子孫節(jié)點(diǎn):<head>節(jié)點(diǎn)和<head>的子節(jié)點(diǎn), ?BeautifulSoup? 有一個(gè)直接子節(jié)點(diǎn)(<html>節(jié)點(diǎn)),卻有很多子孫節(jié)點(diǎn):

len(list(soup.children))
# 1
len(list(soup.descendants))
# 25

.string

如果tag只有一個(gè) ?NavigableString? 類型子節(jié)點(diǎn),那么這個(gè)tag可以使用 .string 得到子節(jié)點(diǎn):

title_tag.string
# u'The Dormouse's story'

如果一個(gè)tag僅有一個(gè)子節(jié)點(diǎn),那么這個(gè)tag也可以使用 ?.string? 方法,輸出結(jié)果與當(dāng)前唯一子節(jié)點(diǎn)的 ?.string? 結(jié)果相同:

head_tag.contents
# [<title>The Dormouse's story</title>]

head_tag.string
# u'The Dormouse's story'

如果tag包含了多個(gè)子節(jié)點(diǎn),tag就無(wú)法確定 ?.string? 方法應(yīng)該調(diào)用哪個(gè)子節(jié)點(diǎn)的內(nèi)容, ?.string? 的輸出結(jié)果是 ?None? :

print(soup.html.string)
# None

.strings 和 stripped_strings

如果tag中包含多個(gè)字符串,可以使用 ?.strings? 來(lái)循環(huán)獲取:

for string in soup.strings:
    print(repr(string))
    # u"The Dormouse's story"
    # u'\n\n'
    # u"The Dormouse's story"
    # u'\n\n'
    # u'Once upon a time there were three little sisters; and their names were\n'
    # u'Elsie'
    # u',\n'
    # u'Lacie'
    # u' and\n'
    # u'Tillie'
    # u';\nand they lived at the bottom of a well.'
    # u'\n\n'
    # u'...'
    # u'\n'

輸出的字符串中可能包含了很多空格或空行,使用 ?.stripped_strings? 可以去除多余空白內(nèi)容:

for string in soup.stripped_strings:
    print(repr(string))
    # u"The Dormouse's story"
    # u"The Dormouse's story"
    # u'Once upon a time there were three little sisters; and their names were'
    # u'Elsie'
    # u','
    # u'Lacie'
    # u'and'
    # u'Tillie'
    # u';\nand they lived at the bottom of a well.'
    # u'...'

全部是空格的行會(huì)被忽略掉,段首和段末的空白會(huì)被刪除

父節(jié)點(diǎn)

繼續(xù)分析文檔樹(shù),每個(gè)tag或字符串都有父節(jié)點(diǎn):被包含在某個(gè)tag中

.parent

通過(guò) ?.parent? 屬性來(lái)獲取某個(gè)元素的父節(jié)點(diǎn).在例子“愛(ài)麗絲”的文檔中,<head>標(biāo)簽是<title>標(biāo)簽的父節(jié)點(diǎn):

title_tag = soup.title
title_tag
# <title>The Dormouse's story</title>
title_tag.parent
# <head><title>The Dormouse's story</title></head>

文檔title的字符串也有父節(jié)點(diǎn):<title>標(biāo)簽

title_tag.string.parent
# <title>The Dormouse's story</title>

文檔的頂層節(jié)點(diǎn)比如<html>的父節(jié)點(diǎn)是 ?BeautifulSoup? 對(duì)象:

html_tag = soup.html
type(html_tag.parent)
# <class 'bs4.BeautifulSoup'>

?BeautifulSoup? 對(duì)象的 ?.parent? 是None:

print(soup.parent)
# None

.parents

通過(guò)元素的 ?.parents? 屬性可以遞歸得到元素的所有父輩節(jié)點(diǎn),下面的例子使用了 ?.parents? 方法遍歷了<a>標(biāo)簽到根節(jié)點(diǎn)的所有節(jié)點(diǎn).

link = soup.a
link
# <a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link1">Elsie</a>
for parent in link.parents:
    if parent is None:
        print(parent)
    else:
        print(parent.name)
# p
# body
# html
# [document]
# None

兄弟節(jié)點(diǎn)

看一段簡(jiǎn)單的例子:

sibling_soup = BeautifulSoup("<a><b>text1</b><c>text2</c></b></a>")
print(sibling_soup.prettify())
# <html>
#  <body>
#   <a>
#    <b>
#     text1
#    </b>
#    <c>
#     text2
#    </c>
#   </a>
#  </body>
# </html>

因?yàn)?lt;b>標(biāo)簽和<c>標(biāo)簽是同一層:他們是同一個(gè)元素的子節(jié)點(diǎn),所以<b>和<c>可以被稱為兄弟節(jié)點(diǎn).一段文檔以標(biāo)準(zhǔn)格式輸出時(shí),兄弟節(jié)點(diǎn)有相同的縮進(jìn)級(jí)別.在代碼中也可以使用這種關(guān)系.

.next_sibling 和 .previous_sibling

在文檔樹(shù)中,使用 ?.next_sibling? 和 ?.previous_sibling? 屬性來(lái)查詢兄弟節(jié)點(diǎn):

sibling_soup.b.next_sibling
# <c>text2</c>

sibling_soup.c.previous_sibling
# <b>text1</b>

<b>標(biāo)簽有 ?.next_sibling? 屬性,但是沒(méi)有 ?.previous_sibling? 屬性,因?yàn)?lt;b>標(biāo)簽在同級(jí)節(jié)點(diǎn)中是第一個(gè).同理,<c>標(biāo)簽有 ?.previous_sibling? 屬性,卻沒(méi)有 ?.next_sibling? 屬性:

print(sibling_soup.b.previous_sibling)
# None
print(sibling_soup.c.next_sibling)
# None

例子中的字符串“text1”和“text2”不是兄弟節(jié)點(diǎn),因?yàn)樗鼈兊母腹?jié)點(diǎn)不同:

sibling_soup.b.string
# u'text1'

print(sibling_soup.b.string.next_sibling)
# None

實(shí)際文檔中的tag的 ?.next_sibling? 和 ?.previous_sibling? 屬性通常是字符串或空白. 看看“愛(ài)麗絲”文檔:

<a  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  class="sister" id="link1">Elsie</a>
<a  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  class="sister" id="link2">Lacie</a>
<a  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  class="sister" id="link3">Tillie</a>

如果以為第一個(gè)<a>標(biāo)簽的 .next_sibling 結(jié)果是第二個(gè)<a>標(biāo)簽,那就錯(cuò)了,真實(shí)結(jié)果是第一個(gè)<a>標(biāo)簽和第二個(gè)<a>標(biāo)簽之間的頓號(hào)和換行符:

link = soup.a
link
# <a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link1">Elsie</a>

link.next_sibling
# u',\n'

第二個(gè)<a>標(biāo)簽是頓號(hào)的 ?.next_sibling? 屬性:

link.next_sibling.next_sibling
# <a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link2">Lacie</a>

.next_siblings 和 .previous_siblings

通過(guò) ?.next_siblings? 和 ?.previous_siblings? 屬性可以對(duì)當(dāng)前節(jié)點(diǎn)的兄弟節(jié)點(diǎn)迭代輸出:

for sibling in soup.a.next_siblings:
    print(repr(sibling))
    # u',\n'
    # <a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link2">Lacie</a>
    # u' and\n'
    # <a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link3">Tillie</a>
    # u'; and they lived at the bottom of a well.'
    # None

for sibling in soup.find(id="link3").previous_siblings:
    print(repr(sibling))
    # ' and\n'
    # <a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link2">Lacie</a>
    # u',\n'
    # <a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link1">Elsie</a>
    # u'Once upon a time there were three little sisters; and their names were\n'
    # None

回退和前進(jìn)

看一下“愛(ài)麗絲” 文檔:

<html><head><title>The Dormouse's story</title></head>
<p class="title"><b>The Dormouse's story</b></p>

HTML解析器把這段字符串轉(zhuǎn)換成一連串的事件: “打開(kāi)<html>標(biāo)簽”,”打開(kāi)一個(gè)<head>標(biāo)簽”,”打開(kāi)一個(gè)<title>標(biāo)簽”,”添加一段字符串”,”關(guān)閉<title>標(biāo)簽”,”打開(kāi)<p>標(biāo)簽”,等等.Beautiful Soup提供了重現(xiàn)解析器初始化過(guò)程的方法.

.next_element 和 .previous_element

?.next_element? 屬性指向解析過(guò)程中下一個(gè)被解析的對(duì)象(字符串或tag),結(jié)果可能與 ?.next_sibling? 相同,但通常是不一樣的.

這是“愛(ài)麗絲”文檔中最后一個(gè)<a>標(biāo)簽,它的 ?.next_sibling? 結(jié)果是一個(gè)字符串,因?yàn)楫?dāng)前的解析過(guò)程因?yàn)楫?dāng)前的解析過(guò)程因?yàn)橛龅搅?lt;a>標(biāo)簽而中斷了:

last_a_tag = soup.find("a", id="link3")
last_a_tag
# <a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link3">Tillie</a>

last_a_tag.next_sibling
# '; and they lived at the bottom of a well.'

但這個(gè)<a>標(biāo)簽的 ?.next_element? 屬性結(jié)果是在<a>標(biāo)簽被解析之后的解析內(nèi)容,不是<a>標(biāo)簽后的句子部分,應(yīng)該是字符串”Tillie”:

last_a_tag.next_element
# u'Tillie'

這是因?yàn)樵谠嘉臋n中,字符串“Tillie” 在分號(hào)前出現(xiàn),解析器先進(jìn)入<a>標(biāo)簽,然后是字符串“Tillie”,然后關(guān)閉</a>標(biāo)簽,然后是分號(hào)和剩余部分.分號(hào)與<a>標(biāo)簽在同一層級(jí),但是字符串“Tillie”會(huì)被先解析.

?.previous_element? 屬性剛好與 ?.next_element? 相反,它指向當(dāng)前被解析的對(duì)象的前一個(gè)解析對(duì)象:

last_a_tag.previous_element
# u' and\n'
last_a_tag.previous_element.next_element
# <a class="sister"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  rel="external nofollow" target="_blank"  id="link3">Tillie</a>

.next_elements 和 .previous_elements

通過(guò) ?.next_elements? 和 ?.previous_elements? 的迭代器就可以向前或向后訪問(wèn)文檔的解析內(nèi)容,就好像文檔正在被解析一樣:

for element in last_a_tag.next_elements:
    print(repr(element))
# u'Tillie'
# u';\nand they lived at the bottom of a well.'
# u'\n\n'
# <p class="story">...</p>
# u'...'
# u'\n'
# None

以上內(nèi)容是否對(duì)您有幫助：

← Beautiful Soup 4 注釋怎么寫(xiě)

Beautiful Soup 4 搜索文檔 →

寫(xiě)筆記

我要補(bǔ)充