對(duì)于Python它是屬于當(dāng)下在人工智能和大數(shù)據(jù)中比較受人喜歡的編程語(yǔ)言,下面我們來(lái)講講“Python怎么爬取網(wǎng)頁(yè)內(nèi)外鏈的代碼?”這個(gè)問(wèn)題的解決方法和相關(guān)內(nèi)容分享!
項(xiàng)目介紹
采用廣度優(yōu)先搜索方法獲取一個(gè)網(wǎng)站上的所有外鏈。
首先,我們進(jìn)入一個(gè)網(wǎng)頁(yè),獲取網(wǎng)頁(yè)的所有內(nèi)鏈和外鏈,再分別進(jìn)入內(nèi)鏈中,獲取該內(nèi)鏈的所有內(nèi)鏈和外鏈,直到訪問(wèn)完所有內(nèi)鏈未知。
代碼大綱
1、用class類定義一個(gè)隊(duì)列,先進(jìn)先出,隊(duì)尾入隊(duì),隊(duì)頭出隊(duì);
2、定義四個(gè)函數(shù),分別是爬取網(wǎng)頁(yè)外鏈,爬取網(wǎng)頁(yè)內(nèi)鏈,進(jìn)入內(nèi)鏈的函數(shù),以及調(diào)函數(shù);
3、爬取百度圖片(https://image.baidu.com/),先定義兩個(gè)隊(duì)列和兩個(gè)數(shù)組,分別來(lái)存儲(chǔ)內(nèi)鏈和外鏈;程序開始時(shí),先分別爬取當(dāng)前網(wǎng)頁(yè)的內(nèi)鏈和外鏈,再分別入隊(duì),對(duì)內(nèi)鏈外鏈進(jìn)行判斷,如果在數(shù)組中沒(méi)有存在,這添加到數(shù)組中;
4、接著調(diào)用deepLinks()函數(shù),采用循環(huán)結(jié)構(gòu),如果當(dāng)前內(nèi)鏈數(shù)量不為空時(shí),則對(duì)存儲(chǔ)內(nèi)鏈的隊(duì)列進(jìn)行出隊(duì),并進(jìn)入該內(nèi)鏈中,再重復(fù)調(diào)用爬取網(wǎng)頁(yè)內(nèi)鏈和網(wǎng)頁(yè)外鏈的函數(shù),進(jìn)行判斷網(wǎng)頁(yè)鏈接是否重復(fù), 不重復(fù)的話,再分別將內(nèi)鏈,外鏈加入到對(duì)應(yīng)的隊(duì)列中,不斷迭代循環(huán);
5、進(jìn)入網(wǎng)頁(yè)內(nèi)所有的內(nèi)鏈,從中搜索出所有的外鏈并且存儲(chǔ)在隊(duì)列中,再輸出。
網(wǎng)站詳情
代碼詳情
隊(duì)列
隊(duì)列是一種特殊的線性表,單向隊(duì)列只能在一端插入數(shù)據(jù)(后),另一端刪除數(shù)據(jù)(前);
它和棧一樣,隊(duì)列是一種操作受限制的線性表;
進(jìn)行插入操作的稱為隊(duì)尾,進(jìn)行刪除操作的稱為隊(duì)頭;
隊(duì)列中的數(shù)據(jù)被稱為元素;沒(méi)有元素的隊(duì)列稱為空隊(duì)列。
由于只能一端刪除或者插入,所以只有最先進(jìn)入隊(duì)列的才能被刪除,因此又被稱為先進(jìn)先出(FIFO—first in first out)線性表。
這里我們用class類定義一個(gè)隊(duì)列,先進(jìn)先出,隊(duì)尾入隊(duì),隊(duì)頭出隊(duì),該隊(duì)列要有定義以下功能:出隊(duì)、入隊(duì)、判斷是否為空、輸出隊(duì)列長(zhǎng)度、返回隊(duì)頭元素。
class Queue(object):
#初始化隊(duì)列
def __init__(self):
self.items = []
#入隊(duì)
def enqueue(self, item):
self.items.append(item)
#出隊(duì)
def dequeue(self):
if self.is_Empty():
print("當(dāng)前隊(duì)列為空??!")
else:
return self.items.pop(0)
#判斷是否為空
def is_Empty(self):
return self.items == []
#隊(duì)列長(zhǎng)度
def size(self):
return len(self.items)
#返回隊(duì)頭元素,如果隊(duì)列為空的話,返回None
def front(self):
if self.is_Empty():
print("當(dāng)前隊(duì)列為空!!")
else:
return self.items[len(self.items) - 1]
內(nèi)鏈外鏈
內(nèi)鏈外鏈的區(qū)別:
內(nèi)鏈:是指同一網(wǎng)站域名下內(nèi)容頁(yè)面之間的互相鏈接。
外鏈:是指在別的網(wǎng)站導(dǎo)入自己網(wǎng)站的鏈接,如友情鏈接、外鏈的搭建等。
通俗的講,內(nèi)鏈即為帶有相同域名的鏈接,而外鏈的域名則不相同。
說(shuō)到內(nèi)鏈外鏈,那必然離不開urllib庫(kù)了,首先導(dǎo)入庫(kù)
from urllib.parse import urlparse
用urlparse模塊來(lái)解析url鏈接,urlparse()模塊將url拆分為6部分:
scheme (協(xié)議) netloc (域名) path (路徑) params (可選參數(shù)) query (連接鍵值對(duì)) fragment (特殊錨)
url='https://image.baidu.com/' a, b = urlparse(url).scheme, urlparse(url).netloc print(a) print(b) #-----------------輸出結(jié)果---------------------# https image.baidu.com
請(qǐng)求頭
Header來(lái)源 用瀏覽器打開需要訪問(wèn)的網(wǎng)頁(yè),按F12,點(diǎn)開network,再按提示按ctr+R,點(diǎn)擊name選擇網(wǎng)站名,再看到有一個(gè)右邊框第一個(gè)headers,找到request headers,這個(gè)就是瀏覽器的請(qǐng)求頭, 復(fù)制其中的user-agent,復(fù)制內(nèi)容。
這里的請(qǐng)求頭為:
headers_={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.68'} html = requests.get(url,headers=headers_)
完整代碼:
class Queue(object):
#初始化隊(duì)列
def __init__(self):
self.items = []
#入隊(duì)
def enqueue(self, item):
self.items.append(item)
#出隊(duì)
def dequeue(self):
if self.is_Empty():
print("當(dāng)前隊(duì)列為空?。?)
else:
return self.items.pop(0)
#判斷是否為空
def is_Empty(self):
return self.items == []
#隊(duì)列長(zhǎng)度
def size(self):
return len(self.items)
#返回隊(duì)頭元素,如果隊(duì)列為空的話,返回None
def front(self):
if self.is_Empty():
print("當(dāng)前隊(duì)列為空??!")
else:
return self.items[len(self.items) - 1]
class Queue(object):
#初始化隊(duì)列
def __init__(self):
self.items = []
#入隊(duì)
def enqueue(self, item):
self.items.append(item)
#出隊(duì)
def dequeue(self):
if self.is_Empty():
print("當(dāng)前隊(duì)列為空?。?)
else:
return self.items.pop(0)
#判斷是否為空
def is_Empty(self):
return self.items == []
#隊(duì)列長(zhǎng)度
def size(self):
return len(self.items)
#返回隊(duì)頭元素,如果隊(duì)列為空的話,返回None
def front(self):
if self.is_Empty():
print("當(dāng)前隊(duì)列為空!!")
else:
return self.items[len(self.items) - 1]
#導(dǎo)入庫(kù)
from urllib.request import urlopen
from urllib.parse import urlparse
from bs4 import BeautifulSoup
import requests
import re
import urllib.parse
import time
import random
queueInt = Queue() #存儲(chǔ)內(nèi)鏈的隊(duì)列
queueExt = Queue() #存儲(chǔ)外鏈的隊(duì)列
externalLinks = []
internalLinks = []
#獲取頁(yè)面中所有外鏈的列表
def getExterLinks(bs, exterurl):
#找出所有以www或http開頭且不包含當(dāng)前URL的鏈接
for link in bs.find_all('a', href = re.compile
('^(http|www)((?!'+urlparse(exterurl).netloc+').)*$')):
#按照標(biāo)準(zhǔn),URL只允許一部分ASCII字符,其他字符(如漢字)是不符合標(biāo)準(zhǔn)的,
#我們的鏈接網(wǎng)址可能存在漢字的情況,此時(shí)就要進(jìn)行編碼。
link.attrs['href'] = urllib.parse.quote(link.attrs['href'],safe='?=&:/')
if link.attrs['href'] is not None:
if link.attrs['href'] not in externalLinks:
queueExt.enqueue(link.attrs['href'])
externalLinks.append(link.attrs['href'])
print(link.attrs['href'])
# return externalLinks
#獲取頁(yè)面中所以內(nèi)鏈的列表
def getInterLinks(bs, interurl):
interurl = '{}://{}'.format(urlparse(interurl).scheme,
urlparse(interurl).netloc)
#找出所有以“/”開頭的內(nèi)部鏈接
for link in bs.find_all('a', href = re.compile
('^(/|.*'+urlparse(interurl).netloc+')')):
link.attrs['href'] = urllib.parse.quote(link.attrs['href'],safe='?=&:/')
if link.attrs['href'] is not None:
if link.attrs['href'] not in internalLinks:
#startsWith()方法用來(lái)判斷當(dāng)前字符串是否是以另外一個(gè)給定的子字符串“開頭”的
if(link.attrs['href'].startswith('//')):
if interurl+link.attrs['href'] not in internalLinks:
queueInt.enqueue(interurl+link.attrs['href'])
internalLinks.append(interurl+link.attrs['href'])
elif(link.attrs['href'].startswith('/')):
if interurl+link.attrs['href'] not in internalLinks:
queueInt.enqueue(interurl+link.attrs['href'])
internalLinks.append(interurl+link.attrs['href'])
else:
queueInt.enqueue(link.attrs['href'])
internalLinks.append(link.attrs['href'])
# return internalLinks
def deepLinks():
num = queueInt.size()
while num > 1:
i = queueInt.dequeue()
if i is None:
break
else:
print('訪問(wèn)的內(nèi)鏈')
print(i)
print('找到的新外鏈')
# html = urlopen(i)
html=requests.get(i,headers=headers_)
time.sleep(random.random()*3)
domain1 = '{}://{}'.format(urlparse(i).scheme, urlparse(i).netloc)
bs = BeautifulSoup(html.content, 'html.parser')
getExterLinks(bs, domain1)
getInterLinks(bs, domain1)
def getAllLinks(url):
global num
# html = urlopen(url)
headers_={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.68'}
html = requests.get(url,headers=headers_)
time.sleep(random.random()*3) #模擬人類行為,間隔隨機(jī)的時(shí)間
domain = '{}://{}'.format(urlparse(url).scheme, urlparse(url).netloc)
bs = BeautifulSoup(html.content, 'html.parser')
getInterLinks(bs, domain)
getExterLinks(bs, domain)
deepLinks()
getAllLinks('https://image.baidu.com/')
爬取結(jié)果
這里我只是截取一部分
數(shù)組中的所有內(nèi)鏈
internalLinks -------------輸出內(nèi)容------------------ ['http://image.baidu.com', 'https://image.baidu.com/img/image/imageplus/index.html?fr=image', 'http://image.baidu.com/search/index?tn=baiduimage&ipn=r&ct=201326592&cl=2&lm=-1&st=-1&fm=result&fr=&sf=1&fmq=1567133149621_R&pv=&ic=0&nc=1&z=0&hd=0&latest=0©right=0&se=1&showtab=0&fb=0&width=&height=&face=0&istype=2&ie=utf-8&sid=&word=%25E5%25A3%2581%25E7%25BA%25B8', 'http://image.baidu.com/search/index?tn=baiduimage&ipn=r&ct=201326592&cl=2&lm=-1&st=-1&fm=result&fr=&sf=1&fmq=1461834053046_R&pv=&ic=0&nc=1&z=&se=1&showtab=0&fb=0&width=&height=&face=0&istype=2&itg=0&ie=utf-8&word=%25E5%25A4%25B4%25E5%2583%258F%23z=0&pn=&ic=0&st=-1&face=0&s=0&lm=-1', 'https://image.baidu.com/search/albumslist?tn=albumslist&word=%25E8%25AE%25BE%25E8%25AE%25A1%25E7%25B4%25A0%25E6%259D%2590&album_tab=%25E8%25AE%25BE%25E8%25AE%25A1%25E7%25B4%25A0%25E6%259D%2590&rn=15&fr=searchindex', 'https://image.baidu.com/search/albumsdetail?tn=albumsdetail&word=%25E5%259F%258E%25E5%25B8%2582%25E5%25BB%25BA%25E7%25AD%2591%25E6%2591%2584%25E5%25BD%25B1%25E4%25B8%2593%25E9%25A2%2598&fr=searchindex_album%2520&album_tab=%25E5%25BB%25BA%25E7%25AD%2591&album_id=7&rn=30', 'https://image.baidu.com/search/albumsdetail?tn=albumsdetail&word=%25E6%25B8%2590%25E5%258F%2598%25E9%25A3%258E%25E6%25A0%25BC%25E6%258F%2592%25E7%2594%25BB&fr=albumslist&album_tab=%25E8%25AE%25BE%25E8%25AE%25A1%25E7%25B4%25A0%25E6%259D%2590&album_id=409&rn=30', 'https://image.baidu.com/search/albumsdetail?tn=albumsdetail&word=%25E7%259A%25AE%25E5%25BD%25B1&fr=albumslist&album_tab=%25E8%25AE%25BE%25E8%25AE%25A1%25E7%25B4%25A0%25E6%259D%2590&album_id=394&rn=30', 'https://image.baidu.com/search/albumsdetail?tn=albumsdetail&word=%25E5%25AE%25A0%25E7%2589%25A9%25E5%259B%25BE%25E7%2589%2587&fr=albumslist&album_tab=%25E5%258A%25A8%25E7%2589%25A9&album_id=688&rn=30', 'https://image.baidu.com/search/albumsdetail?tn=albumsdetail&word=%25E8%2588%25AA%25E6%258B%258D%25E5%259C%25B0%25E7%2590%2583%25E7%25B3%25BB%25E5%2588%2597&fr=albumslist&album_tab=%25E8%25AE%25BE%25E8%25AE%25A1%25E7%25B4%25A0%25E6%259D%2590&album_id=312&rn=30', 'https://image.baidu.com/search/albumslist?tn=albumslist&word=%25E4%25BA%25BA%25E7%2589%25A9&album_tab=%25E4%25BA%25BA%25E7%2589%25A9&rn=15&fr=searchindex_album', 'http://image.baidu.com/static/html/advanced.html', 'https://image.baidu.com/', 'http://image.baidu.com/']
數(shù)組中的所有外鏈
externalLinks -------------輸出內(nèi)容------------------ ['http://news.baidu.com/', 'https://www.hao123.com/', 'http://map.baidu.com/', 'https://haokan.baidu.com/?sfrom=baidu-top/', 'http://tieba.baidu.com/', 'https://xueshu.baidu.com/', 'http://www.baidu.com/more/', 'https://pan.baidu.com', 'https://zhidao.baidu.com', 'https://baike.baidu.com', 'https://baobao.baidu.com', 'https://wenku.baidu.com', 'https://jingyan.baidu.com', 'http://music.taihe.com', 'https://www.baidu.com', 'https://www.baidu.com/', 'http://www.baidu.com/duty/', 'http://www.baidu.com/search/image_help.html', 'http://www.beian.gov.cn/portal/registerSystemInfo?recordcode=11000002000001', 'http://help.baidu.com/question', 'http://www.baidu.com/search/jubao.html', 'http://www.baidu.com/search/faq_image.html%2305']
那么以上的內(nèi)容就是今天要和大家分享的有關(guān)于:“Python怎么爬取網(wǎng)頁(yè)內(nèi)外鏈的代碼?”這個(gè)問(wèn)題的全部?jī)?nèi)容,更多的相關(guān)知識(shí)和內(nèi)容我們都可以在w3cschool中進(jìn)行學(xué)習(xí)和了解。