Python爬虫框架之scrapy如何使用scrapy-splash爬虫动态js页面

旧城等待， 2021-09-18 23:18 552阅读 0赞

首先splash环境搭建：[https://blog.csdn.net/weixin\_43343144/article/details/88022756][https_blog.csdn.net_weixin_43343144_article_details_88022756]

安装splash可能报的错误：[https://blog.csdn.net/weixin\_43343144/article/details/89305941][https_blog.csdn.net_weixin_43343144_article_details_89305941]

pipenv install scrapy-splash

splash官方文档：[https://splash.readthedocs.io/en/stable/api.html\#render-html][https_splash.readthedocs.io_en_stable_api.html_render-html]

scrapy-splash官方文档：[https://github.com/scrapy-plugins/scrapy-splash][https_github.com_scrapy-plugins_scrapy-splash]

别人翻译的scrapy-splash文章：[https://www.e-learn.cn/content/qita/800748][https_www.e-learn.cn_content_qita_800748]

## ##

### **scrapy-splash使用总结【使用SplashRequest代替了Request就可以抓取动态js页面了。SplashRequest就是继承了Request类增加了一些配置动态js的参数，使用起来和抓取静态页面原理是一样的，就是多了一个splash动态解析！】** ###

### **settings.py配置文件：** ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzM0MzE0NA_size_16_color_FFFFFF_t_70][]

SPLASH_URL ='http://192.168.99.100:8050'
    
    //在settings文件DOWNLOADER_MIDDLEWARES选项中增加下面三个
    DOWNLOADER_MIDDLEWARES = {
        'scrapy_splash.SplashCookiesMiddleware': 723,
        'scrapy_splash.SplashMiddleware': 725,
        'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
    }
    
    //在settings文件SPIDER_MIDDLEWARES选项中增加下面这个
    SPIDER_MIDDLEWARES = {
        'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
    }
    
    DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
    HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

### **quotes.py爬虫文件：** ###

# -*- coding: utf-8 -*-
    import scrapy
    from scrapy_splash import SplashRequest,SplashResponse
    from scrapy.http import Request,Response
    
    
    class QuotesSpider(scrapy.Spider):
        name = 'quotes'
        allowed_domains = ['quotes.toscrape.com']
        # start_urls = ['http://quotes.toscrape.com/js/']
        url = "http://quotes.toscrape.com/js"
    
    
        def start_requests(self):
            # 利用SplashRequest对象直接可以读取动态js页面(前提必须搭建好了splash环境)
            # SplashRequest其实就是继承了Request,并实现了splash的集成！
            yield SplashRequest(url=self.url,args={"images":0,"timeout":30})
    
    
        def parse(self, response):
            res = response #type:Response
            quotes = res.css("div.quote .text::text").getall()
            pass

[https_blog.csdn.net_weixin_43343144_article_details_88022756]: https://blog.csdn.net/weixin_43343144/article/details/88022756
[https_blog.csdn.net_weixin_43343144_article_details_89305941]: https://blog.csdn.net/weixin_43343144/article/details/89305941
[https_splash.readthedocs.io_en_stable_api.html_render-html]: https://splash.readthedocs.io/en/stable/api.html#render-html
[https_github.com_scrapy-plugins_scrapy-splash]: https://github.com/scrapy-plugins/scrapy-splash
[https_www.e-learn.cn_content_qita_800748]: https://www.e-learn.cn/content/qita/800748
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzM0MzE0NA_size_16_color_FFFFFF_t_70]: /images/20210812/3056af837e6a48a783e62e4671d2a34e.png