Scrapy第十一(③)篇：selenium4模拟器-抓取Ajax请求（单线程）

太过爱你忘了你带给我的痛 2024-04-08 09:56 134阅读 0赞

在Vue、React横行的时代，前后端分离使用Ajax做交互的比比皆是。Ajax请求往往带有各种加密验证，解密起来费力气。最好是selenium直接拦截ajax的结果。

#### 1.安装依赖 ####

pip install browsermob-proxy

#### 2.下载代理驱动 ####

[https://github.com/lightbody/browsermob-proxy/releases/tag/browsermob-proxy-2.1.4![icon-default.png_t_M666][]https://github.com/lightbody/browsermob-proxy/releases/tag/browsermob-proxy-2.1.4][https_github.com_lightbody_browsermob-proxy_releases_tag_browsermob-proxy-2.1.4_icon-default.png_t_M666_https_github.com_lightbody_browsermob-proxy_releases_tag_browsermob-proxy-2.1.4]

#### ![8f377660ddd14cfc8014c6a18fa7fba2.png][]3.解压并放到项目目录中。 ####

![e62462816603482fb64e04cf0cd7684b.png][]

#### 4.测试用例 ####

import time
    
    from selenium import webdriver
    from selenium.webdriver.chrome.service import Service as ChromeService
    from webdriver_manager.chrome import ChromeDriverManager
    
    from browsermobproxy import Server
    
    if __name__ == '__main__':
    
        # 开启代理
        BMPserver = Server(r'D:\workspace\scrapyproject\testscrapy\browsermob-proxy-2.1.4\bin\browsermob-proxy.bat')
        BMPserver.start()
        BMPproxy = BMPserver.create_proxy()
    
        # 初始化驱动
        service = ChromeService(executable_path=ChromeDriverManager().install())
    
        # 配置代理参数
        options = webdriver.ChromeOptions()
        options.add_argument('--ignore-certificate-errors')
        options.add_argument('--proxy-server={}'.format(BMPproxy.proxy))
    
        # 获取浏览器实例
        driver = webdriver.Chrome(service=service, options=options)
    
        # 监听ajax请求
        BMPproxy.new_har("lagou", options={'captureContent': True, 'captureContent': True})
    
        # 访问网页
        driver.get("https://dynamic2.scrape.center/")
    
        # 等待ajax加载完毕
        time.sleep(10)
    
        # 获取ajax加载结果
        result = BMPproxy.har
        for entry in result['log']['entries']:
            entry_url = entry['request']['url']
            # 根据URL找到你想要的的ajax请求。

#### 5.做成scrapy中间件 ####

class SeleniumMiddleware(object):
        def process_response(self, request, response, spider):
            # 开启代理
            BMPserver = Server(r'D:\workspace\scrapyproject\testscrapy\browsermob-proxy-2.1.4\bin\browsermob-proxy.bat')
            BMPserver.start()
            BMPproxy = BMPserver.create_proxy()
    
            # 初始化驱动
            service = ChromeService(executable_path=ChromeDriverManager().install())
            # 配置
            options = webdriver.ChromeOptions()
            options.add_argument('--ignore-certificate-errors')
            options.add_argument('--proxy-server={}'.format(BMPproxy.proxy))
            # 无头模式
            # options.add_argument('--headless')
            # 获取浏览器实例
            driver = webdriver.Chrome(service=service, options=options)
    
            # 监听ajax请求
            BMPproxy.new_har("lagou", options={'captureContent': True, 'captureContent': True})
    
            # 访问网页
            driver.get(request.url)
            # 等待ajax加载完毕
            time.sleep(5)
    
            # 获取ajax加载结果
            result = BMPproxy.har
            for entry in result['log']['entries']:
                entry_url = entry['request']['url']
                # 根据URL找到你想要的的ajax请求。
            driver.quit()
            return scrapy.http.HtmlResponse(url=request.url, body=json.dumps(result).encode('utf-8'), encoding='utf-8', request=request)

#### 6.开启中间件 ####

DOWNLOADER_MIDDLEWARES = {
        'testscrapy.middlewares.SeleniumMiddleware': 543,
    }  # 下载中间件

[icon-default.png_t_M666]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/994958791e214c75a626792000e0e8db.png
[https_github.com_lightbody_browsermob-proxy_releases_tag_browsermob-proxy-2.1.4_icon-default.png_t_M666_https_github.com_lightbody_browsermob-proxy_releases_tag_browsermob-proxy-2.1.4]: https://github.com/lightbody/browsermob-proxy/releases/tag/browsermob-proxy-2.1.4
[8f377660ddd14cfc8014c6a18fa7fba2.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/0c306794995c4b50937c2a110c8e5881.png
[e62462816603482fb64e04cf0cd7684b.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/2103d6d005b04293804a12c60d237193.png