scrapy实战——利用CrawlSpider爬取腾讯社招全部岗位信息（进行有一定深度的爬取）

港控/mmm° 2022-06-01 13:21 179阅读 0赞

经过scrapy的简单学习，我们实现这样一个爬虫：爬取腾讯社招的全部岗位信息，将粗略的大致信息保存在tencent.json文件中，将岗位的进一步具体信息（职责、要求）保存在positiondescribe.json文件中。  
即，我们需要两个item进行页面信息的保存，同时要继承CrawlSpider对页面链接进行相应提取。  
项目目录如下：(创建名为TencntSpider的项目)

TencentSpider
        │  items.py
        │  middlewares.py
        │  pipelines.py
        │  settings.py
        │  __init__.py
        │
        ├─spiders
        │  │  tencent.py
        │  │  __init__.py
        │  │
        │  └─__pycache__
        │          tencent.cpython-36.pyc
        │          __init__.cpython-36.pyc
        │
        └─__pycache__
                items.cpython-36.pyc
                pipelines.cpython-36.pyc
                settings.cpython-36.pyc
                __init__.cpython-36.pyc

难点主要在：  
1. 对多个item的处理：在pipelines文件中对传入的item做判断：利用**class**.**name**的方法可以对类名进行判断！  
2. 对于spider中rules的书写，要清楚我们需要过滤的链接或页面。  
3. 对于爬虫文件中parse方法的书写：要记得一旦继承了CrawlSpider类便不能再重写parse方法，我们要自己编写parse方法，因为我们提取了链接之后，要对链接进行跟进处理，进入详细信息的页面，所以我们要写两个parse方法！  
items.py代码如下：

# items.py
    # -*- coding: utf-8 -*-
    
    # Define here the models for your scraped items
    #
    # See documentation in:
    # https://doc.scrapy.org/en/latest/topics/items.html
    
    import scrapy
    
    
    class TencentspiderItem(scrapy.Item):
        # 职位名
        positionName = scrapy.Field()
        # 详情链接
        positionLink = scrapy.Field()
        # 职位类别
        positionType = scrapy.Field()
        # 招聘人数
        peopleNum = scrapy.Field()
        # 工作地点
        workLocation = scrapy.Field()
        # 发布时间
        publishTime = scrapy.Field()
    
    class PositionDescribe(scrapy.Item):
        # 职位名
        positionName = scrapy.Field()
        # 职位类别
        positionType = scrapy.Field()
        # 招聘人数
        peopleNum = scrapy.Field()
        # 工作地点
        workLocation = scrapy.Field()
        # 职责
        duty = scrapy.Field()
        # 要求
        requirement = scrapy.Field()

tencent.py代码如下：

# tencent.py
    # -*- coding: utf-8 -*-
    from scrapy.linkextractors import LinkExtractor
    from scrapy.spiders import CrawlSpider, Rule
    from TencentSpider.items import TencentspiderItem
    from TencentSpider.items import PositionDescribe
    
    
    class TencentSpider(CrawlSpider):
        name = 'tencent'
        allowed_domains = ['hr.tencent.com']
        start_urls = ['https://hr.tencent.com/position.php?&start=0#a']
    
        rules = (
            Rule(LinkExtractor(allow=r'&start=\d+'), callback='tencentParse', follow=True),
            Rule(LinkExtractor(allow=r'/position_detail.php?'), callback='positionParse', follow=True)
        )
    
        def tencentParse(self, response):
            jobs_list = response.xpath('//tr[@class="even"or@class="odd"]')
    
            for node in jobs_list:
                item = TencentspiderItem()
                name = node.xpath('./td[1]/a/text()').extract()[0]
                link = node.xpath('./td[1]/a/@href').extract()[0]
                type = ''.join(node.xpath('./td[2]/text()').extract())
                num = node.xpath('./td[3]/text()').extract()[0]
                location = node.xpath('./td[4]/text()').extract()[0]
                date = node.xpath('./td[5]/text()').extract()[0]
                item['positionName'] = name
                item['positionLink'] = 'https://hr.tencent.com/' + str(link)
                item['positionType'] = type
                item['peopleNum'] = num
                item['workLocation'] = location
                item['publishTime'] = date
    
                yield item
    
        def positionParse(self, response):
            item = PositionDescribe()
            name = response.xpath('//td[@id="sharetitle"]/text()').extract()
            location = response.xpath('//tr[@class="c bottomline"]/td[1]/text()').extract()
            type = response.xpath('//tr[@class="c bottomline"]/td[2]/text()').extract()
            num = response.xpath('//tr[@class="c bottomline"]/td[3]/text()').extract()
            s = ''
            duties = response.xpath('//table//tr[3]//ul/li/text()').extract()
            for duty in duties:
                s += duty
            requirements = response.xpath('//table//tr[4]//ul/li/text()').extract()
            q = ''
            for require in requirements:
                q += require
    
            # 职位名
            item['positionName'] = name
            # 职位类别
            item['positionType'] = type
            # 招聘人数
            item['peopleNum'] = num
            # 工作地点
            item['workLocation'] = location
            # 职责
            item['duty'] = s
            # 要求
            item['requirement'] = q
    
            yield item

pipelines.py代码如下：

# pipelines.py
    # -*- coding: utf-8 -*-
    
    # Define your item pipelines here
    #
    # Don't forget to add your pipeline to the ITEM_PIPELINES setting
    # See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
    import json
    
    class TencentspiderPipeline(object):
    
        def __init__(self):
            # isinstance
            self.file = open('tencent.json', 'a', encoding='utf-8')
            self.file2 = open('positiondescribe.json', 'a', encoding='utf-8')
    
        def process_item(self, item, spider):
            if item.__class__.__name__ == 'TencentspiderItem':
                jsontext = json.dumps(dict(item), ensure_ascii=False) + ',\n'
                self.file.write(jsontext)
            else:
                jsontext = json.dumps(dict(item), ensure_ascii=False) + ',\n'
                self.file2.write(jsontext)
    
            return item
    
        def close_spider(self, spider):
            self.file.close()
            self.file2.close()

主要便是这三个文件，同时爬取之后，会生成两个json文件。