python 爬虫 1 （urllib、requests的使用、代理服务器）

我会带着你远行 2023-10-08 12:04 148阅读 0赞

#### urllib、requests的使用 ####

*   *  写在前面
     *  1、urllib的使用
     *   *  1.1、request.urlopen
         *  1.2、使用GET参数
         *  1.3、使用POST传递参数
         *  1.4、代理ip请求远程url
         *  1.5、urlpase模块
     *  2、requests的使用
     *   *  2.1、requests.get()
         *  2.2、requests.post()
         *  2.3、自定义headers
         *  2.4、带cookies请求
         *  2.5、Basic-auth认证请求
         *  2.6、主动抛出状态码异常
         *  2.7、session（）
         *  2.8、代理
     *  3、代理服务器

### 写在前面 ###

urllib：它是 Python 内置的 HTTP 请求库  
Requests：也是HTTP 请求库，比urllib更易使用

Requests中文翻译文档

> http://2.python-requests.org/zh\_CN/latest/user/quickstart.html

### 1、urllib的使用 ###

urllib的request模块可以非常方便地抓取URL内容

#### 1.1、request.urlopen ####

①例如，对豆瓣的一个URLhttps://api.douban.com/v2/book/2129650进行抓取，并返回响应：

from urllib import request
    
    with request.urlopen('https://api.douban.com/v2/book/2129650') as f:
        data = f.read()
        print('Status:', f.status, f.reason)
        for k, v in f.getheaders():
            print('%s: %s' % (k, v))
        print('Data:', data.decode('utf-8'))

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70]

② req.add\_header ()例：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 1]

<table> 
 <thead> 
  <tr> 
   <th>urllib的方法、属性</th> 
   <th>解释</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>f.read()</td> 
   <td>读取f的内容</td> 
  </tr> 
  <tr> 
   <td>f.status</td> 
   <td>状态码</td> 
  </tr> 
  <tr> 
   <td>f.reason</td> 
   <td>响应消息</td> 
  </tr> 
  <tr> 
   <td>f.getheaders()</td> 
   <td>获取f的头信息</td> 
  </tr> 
  <tr> 
   <td>req.add_header ()</td> 
   <td>添加访问头信息</td> 
  </tr> 
 </tbody> 
</table>

#### 1.2、使用GET参数 ####

![在这里插入图片描述][20200303170559574.png]

#### 1.3、使用POST传递参数 ####

![在这里插入图片描述][20200303172630919.png]

#### 1.4、代理ip请求远程url ####

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 2]

#### 1.5、urlpase模块 ####

urlparse模块主要是用于解析url中的参数 对url按照一定格式进行 拆分或拼接  
![在这里插入图片描述][2020030317294777.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 3]

### 2、requests的使用 ###

安装

pip install requests

#### 2.1、requests.get() ####

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 4]  
get请求带参数  
![在这里插入图片描述][20200303183310163.png]

#### 2.2、requests.post() ####

![在这里插入图片描述][20200303183210525.png]  
requests其他请求方式

>>> r = requests.put('http://httpbin.org/put', data = {
        'key':'value'})
    >>> r = requests.delete('http://httpbin.org/delete')
    >>> r = requests.head('http://httpbin.org/get')
    >>> r = requests.options('http://httpbin.org/get')

#### 2.3、自定义headers ####

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 5]

#### 2.4、带cookies请求 ####

![在这里插入图片描述][20200303183623268.png]

#### 2.5、Basic-auth认证请求 ####

![在这里插入图片描述][20200303184222293.png]

#### 2.6、主动抛出状态码异常 ####

![在这里插入图片描述][20200303184710501.png]

#### 2.7、session（） ####

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 6]

#### 2.8、代理 ####

![在这里插入图片描述][20200303185423358.png]

### 3、代理服务器 ###

> 这里我用文字加截图的形式，如果想要简介一点的教程可以参考  
> https://blog.csdn.net/weixin\_41607151/article/details/80671649

用Xshell打开服务器，这里我服务器的系统是linux centos7

安装 tinyproxy:

yum install tinyproxy

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 7]  
安装中间可能会问你yes or no 输入一个y就行

装完之后编辑配置文件：vim /etc/tinyproxy/tinyproxy.conf

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 8]  
使用vim的搜索功能，搜Allow：/Allow  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 9]  
显示行数：输入“:set number”回车  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 10]  
按键盘上的n可以向下查找Allow，这行一般在210行这儿  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 11]  
按键盘上的insert，然后输入一个\#，把这行注释掉  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 12]  
找到23行的port  
这里我的安全策略里面打开的是41801端口，所以我改成41801，然后保存并退出  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 13]  
不知道安全策略端口的看下图  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 14]  
重启服务：systemctl restart tinyproxy  
![在这里插入图片描述][20200303201908381.png]  
打开日志：tail -f /var/log/tinyproxy/tinyproxy.log  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 15]  
使用之前先查看防火墙是否开放41801端口

firewall-cmd --zone=public --list-ports 查询开放状态
    firewall-cmd --zone=public --add-port=41801/tcp --permanent  设置41801开放
    firewall-cmd --reload

liunix环境下使用代理

![在这里插入图片描述][2020030321215596.png]  
windows下使用代理  
![在这里插入图片描述][20200303212554600.png]  
![在这里插入图片描述][20200303212131407.png]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200303163454284.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200303164739606.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[20200303170559574.png]: https://img-blog.csdnimg.cn/20200303170559574.png
[20200303172630919.png]: https://img-blog.csdnimg.cn/20200303172630919.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200303172815714.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[2020030317294777.png]: https://img-blog.csdnimg.cn/2020030317294777.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200303173136306.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200303183030560.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[20200303183310163.png]: https://img-blog.csdnimg.cn/20200303183310163.png
[20200303183210525.png]: https://img-blog.csdnimg.cn/20200303183210525.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200303183522736.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[20200303183623268.png]: https://img-blog.csdnimg.cn/20200303183623268.png
[20200303184222293.png]: https://img-blog.csdnimg.cn/20200303184222293.png
[20200303184710501.png]: https://img-blog.csdnimg.cn/20200303184710501.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200303185135986.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[20200303185423358.png]: https://img-blog.csdnimg.cn/20200303185423358.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200303193410538.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200303193716643.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200303194057995.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20200303194210482.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200303194351883.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20200303200630484.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20200303201342850.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20200303201559418.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[20200303201908381.png]: https://img-blog.csdnimg.cn/20200303201908381.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/20200303202106464.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FfX2ludF9f,size_16,color_FFFFFF,t_70
[2020030321215596.png]: https://img-blog.csdnimg.cn/2020030321215596.png
[20200303212554600.png]: https://img-blog.csdnimg.cn/20200303212554600.png
[20200303212131407.png]: https://img-blog.csdnimg.cn/20200303212131407.png