Selenium+Python实现百度热搜数据的获取

Selenium+Python实现百度热搜数据的获取

一，准备环境

• pycharm 2022.2

• python 3.10

• chromedriver

• selenium 4.3.0

• pymysql 0.7.11

• mysql 5.7

二，导入项目模块

import os
import pymysql
import datetime
from selenium import webdriver
from selenium.webdrivermon.by import By

三, 测试百度热搜接口是否通畅

url = '=realtime'
res = (url)
print(res.status_code)

返回状态码为200即为接口通畅

四，进入百度热搜

options = webdriver.ChromeOptions()
options.add_experimental_option('useAutomationExtension', False)
options.add_experimental_option('excludeSwitches', ['enable-automation'])
options.add_argument('--start-maximized')
options.add_argument('disable-infobars')
drivers = webdriver.Chrome(options=options)
('=realtime')

五, 获取百度热搜数据

tittleList = []
exponentList = []
for tittleIndex in range(1, 32):drivers.implicitly_wait(10)xpath = '//*[@id="sanRoot"]/main/div[2]/div/div[2]/div[' + str(tittleIndex) + ']/div[2]/a/div[1]'tittle = drivers.find_element(By.XPATH, xpath).texttittleList.append(tittle)
for exponentIndex in range(1, 32):drivers.implicitly_wait(10)xpath = '//*[@id="sanRoot"]/main/div[2]/div/div[2]/div[' + str(exponentIndex) + ']/div[1]/div[2]'exponent = drivers.find_element(By.XPATH, xpath).textexponentList.append(exponent)

六，链接数据库

6.1 创建数据库表

 CREATE TABLE baidu_info(id int NOT NULL AUTO_INCREMENT,tittle varchar(150),exponent varchar(128),PRIMARY KEY(id))ENGINE=InnoDB;

6.2 python链接写入数据库

config = {'host': '127.0.0.1','user': 'root','password': 'root','database': 'baidu'
}
db = t(**config)
cursor = db.cursor()
for index, element in enumerate(tittleList):sql = 'INSERT INTO baidu_info(tittle,exponent) VALUES("' + tittleList[index] + '","' + exponentList[index] + '")'print(ute(sql)dbmit()

7.完活收工