集成学习：Bagging、随机森林、Boosting、GBDT

曾经终败给现在 2023-07-09 02:03 42阅读 0赞

# ![20191009191333910.png][][日萌社][Link 1] #

[人工智能AI：Keras PyTorch MXNet TensorFlow PaddlePaddle 深度学习实战（不定时更新）][AI_Keras PyTorch MXNet TensorFlow PaddlePaddle]

--------------------

[集成学习：Bagging、随机森林、Boosting、GBDT][Bagging_Boosting_GBDT]

[集成学习：XGBoost][XGBoost]

[集成学习：lightGBM（一）][lightGBM]

[集成学习：lightGBM（二）][lightGBM 1]

--------------------

# 5.1 集成学习算法简介 #

## 1 什么是集成学习 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70][]

集成学习通过建立几个模型来解决单一预测问题。它的工作原理是**生成多个分类器/模型**，各自独立地学习和作出预测。**这些预测最后结合成组合预测，因此优于任何一个单分类的做出预测。**

## 2 **复习：机器学习的两个核心任务** ##

*  任务一：**如何优化训练数据** —> 主要用于**解决欠拟合问题**
 *  任务二：**如何提升泛化性能** —> 主要用于**解决过拟合问题**

## 3 集成学习中boosting和Bagging ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 1][]

**只要单分类器的表现不太差，集成学习的结果总是要好于单分类器的**

## 4 小结 ##

*  什么是集成学习【了解】
    
     *  通过建立几个模型来解决单一预测问题
 *  机器学习两个核心任务【知道】
    
     *  1.解决欠拟合问题
        
         *  弱弱组合变强
         *  boosting
     *  2.解决过拟合问题
        
         *  互相遏制变壮
         *  Bagging

--------------------

# 5.2 Bagging和随机森林 #

## 1 Bagging集成原理 ##

目标：把下面的圈和方块进行分类

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 2][]

实现过程：

1) 采样不同数据集

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 3][]

2)训练分类器

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 4][]

3)平权投票，获取最终结果

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 5][]

4)主要实现过程小结

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 6][]

## 2 随机森林构造过程 ##

在机器学习中，**随机森林是一个包含多个决策树的分类器**，并且其输出的类别是由个别树输出的类别的众数而定。

**随机森林** **= Bagging +** **决策树**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 7][]

例如, 如果你训练了5个树, 其中有4个树的结果是True, 1个树的结果是False, 那么最终投票结果就是True

**随机森林够造过程中的关键步骤**(M表示特征数目)：

**1)一次随机选出一个样本，有放回的抽样，重复N次(有可能出现重复的样本)**

**2) 随机去选出m个特征, m <<M，建立决策树**

*  **思考**
    
     *  1.为什么要随机抽样训练集？　　
        
         *  如果不进行随机抽样，每棵树的训练集都一样，那么最终训练出的树分类结果也是完全一样的
     *  2.为什么要有放回地抽样？
        
         *  如果不是有放回的抽样，那么每棵树的训练样本都是不同的，都是没有交集的，这样每棵树都是“有偏的”，都是绝对“片面的”(当然这样说可能不对)，也就是说每棵树训练出来都是有很大的差异的；而随机森林最后分类取决于多棵树(弱分类器)的投票表决。

## 3 包外估计 (Out-of-Bag Estimate) ##

在随机森林构造过程中，如果进行有放回的抽样，我们会发现，总是有一部分样本我们选不到。

*  这部分数据，占整体数据的比重有多大呢？
 *  这部分数据有什么用呢？

### 3.1 包外估计的定义 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 8][]

![20200227085732837.png][]

# 如何理解无偏估计？无偏估计有什么用？ #

## 1.如何理解无偏估计 ##

**无偏估计**：就是我认为**所有样本出现的概率一样**。

假如有N种样本我们认为所有样本出现概率都是1/N。然后根据这个来计算数学期望。此时的数学期望就是我们平常讲的平均值。

> 数学期望本质就是平均值

## 2.无偏估计为何叫做“无偏”？它要“估计”什么？ ##

首先回答第一个问题：它要“估计”什么？

*  它要估计的是整体的数学期望（平均值）。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 9][]

## 3.为何要用无偏估计？ ##

因为现实生活中我不知道某个样本出现的概率啊，就像骰子，我不知道他是不是加过水银。

所以我们暂时按照每种情况出现概率一样来算。

### 3.2 包外估计的用途 ###

*  当基学习器是决策树时，可使用包外样本来辅助剪枝 ，或用于估计决策树中各结点的后验概率以辅助对零训练样本结点的处理；
 *  当基学习器是神经网络时，可使用包外样本来辅助早期停止以减小过拟合 。

## 3 随机森林api介绍 ##

*  sklearn.ensemble.RandomForestClassifier(n\_estimators=10, criterion=’gini’, max\_depth=None, bootstrap=True, random\_state=None, min\_samples\_split=2)
    
     *  **n\_estimators**：integer，optional(default = 10)森林里的树木数量120,200,300,500,800,1200
        
         *  在利用最大投票数或平均值来预测之前，你想要建立子树的数量。
     *  **Criterion**：string，可选(default =“gini”)
        
         *  分割特征的测量方法
     *  **max\_depth**：integer或None，可选(默认=无)
        
         *  树的最大深度 5,8,15,25,30
     *  **max\_features="auto”**,每个决策树的最大特征数量
        
         *  If "auto", then `max_features=sqrt(n_features)`.
         *  If "sqrt", then `max_features=sqrt(n_features)`(same as "auto").
         *  If "log2", then `max_features=log2(n_features)`.
         *  If None, then `max_features=n_features`.
     *  **bootstrap**：boolean，optional(default = True)
        
         *  是否在构建树时使用放回抽样
     *  **min\_samples\_split** 内部节点再划分所需最小样本数
        
         *  这个值限制了子树继续划分的条件，如果某节点的样本数少于min\_samples\_split，则不会继续再尝试选择最优特征来进行划分，默认是2。
         *  如果样本量不大，不需要管这个值。如果样本量数量级非常大，则推荐增大这个值。
     *  **min\_samples\_leaf** 叶子节点的最小样本数
        
         *  这个值限制了叶子节点最少的样本数，如果某叶子节点数目小于样本数，则会和兄弟节点一起被剪枝， 默认是1。
         *  叶是决策树的末端节点。 较小的叶子使模型更容易捕捉训练数据中的噪声。
         *  > 一般来说，我更偏向于将最小叶子节点数目设置为大于50。
     *  **min\_impurity\_split:** 节点划分最小不纯度
        
         *  这个值限制了决策树的增长，如果某节点的不纯度(基于基尼系数，均方差)小于这个阈值，则该节点不再生成子节点。即为叶子节点 。
         *  > 一般不推荐改动默认值1e-7。
 *  上面决策树参数中最重要的包括
    
     *  **最大特征数max\_features，**
     *  **最大深度max\_depth，**
     *  **内部节点再划分所需最小样本数min\_samples\_split**
     *  **叶子节点最少样本数min\_samples\_leaf。**

## 4 随机森林预测案例 ##

*  实例化随机森林

# 随机森林去进行预测
    rf = RandomForestClassifier()

*  定义超参数的选择列表

param = {"n_estimators": [120,200,300,500,800,1200], "max_depth": [5, 8, 15, 25, 30]}

*  使用GridSearchCV进行网格搜索

# 超参数调优
    gc = GridSearchCV(rf, param_grid=param, cv=2)
    
    gc.fit(x_train, y_train)
    
    print("随机森林预测的准确率为：", gc.score(x_test, y_test))

> 注意
> 
>  *  随机森林的建立过程
>  *  树的深度、树的个数等需要进行超参数调优

## 5 bagging集成优点 ##

**Bagging + 决策树/线性回归/逻辑回归/深度学习… = bagging集成学习方法**

经过上面方式组成的集成学习方法:

1.  **均可在原有算法上提高约2%左右的泛化正确率**
2.  **简单, 方便, 通用**

## 6 小结 ##

*  bagging集成过程【知道】
    
     *  1.采样 — 从所有样本里面，采样一部分
     *  2.学习 — 训练弱学习器
     *  3.集成 — 使用平权投票
 *  随机森林介绍【知道】
    
     *  随机森林定义
        
         *  随机森林 = Bagging + 决策树
     *  流程：
        
         *  1.随机选取m条数据
         *  2.随机选取k个特征
         *  3.训练决策树
         *  4.重复1-3
         *  5.对上面的若决策树进行平权投票
     *  注意：
        
         *  1.随机选取样本，且是有放回的抽取
         *  2.选取特征的时候吗，选择m<<M
         *  M是所有的特征数
     *  包外估计
        
         *  如果进行有放回的对数据集抽样，会发现，总是有一部分样本选不到；
     *  api
        
         *  sklearn.ensemble.RandomForestClassifier()
 *  Bagging + 决策树/线性回归/逻辑回归/深度学习… = bagging集成学习方法【了解】
 *  bagging的优点【了解】
    
     *  1.均可在原有算法上提高约2%左右的泛化正确率
     *  2.简单, 方便, 通用

--------------------

# 5.3 otto案例介绍 -- Otto Group Product Classification Challenge #

## 1.背景介绍 ##

奥托集团是世界上最大的电子商务公司之一，在20多个国家设有子公司。该公司每天都在世界各地销售数百万种产品,所以对其产品根据性能合理的分类非常重要。

不过,在实际工作中,工作人员发现,许多相同的产品得到了不同的分类。本案例要求,你对奥拓集团的产品进行正确的分分类。尽可能的提供分类的准确性。

链接：[https://www.kaggle.com/c/otto-group-product-classification-challenge/overview][https_www.kaggle.com_c_otto-group-product-classification-challenge_overview]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 10][]

## 2.数据集介绍 ##

*  本案例中，数据集包含大约200,000种产品的93个特征。
 *  其目的是建立一个能够区分otto公司主要产品类别的预测模型。
 *  所有产品共被分成九个类别（例如时装，电子产品等）。

![20200227085945233.png][]

*  **id** - 产品id
 *  **feat\_1, feat\_2, ..., feat\_93** - 产品的各个特征
 *  **target** - 产品被划分的类别

## 3.评分标准 ##

本案例中，最后结果使用多分类对数损失进行评估。

具体公式：

![20200227090006970.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 11][]

## 4.实现过程 ##

### 4.1 流程分析 ###

*  获取数据
 *  数据基本处理
    
     *  数据量比较大，尝试是否可以进行数据分割
     *  转换目标值表示方式
 *  模型训练
    
     *  模型基本训练

### 4.2 代码实现 ###

In \[1\]:

import numpy as np  
import pandas as pd  
import matplotlib.pyplot as plt

数据获取

In \[2\]:

data = pd.read\_csv("./data/otto/train.csv")

In \[3\]:

data.head()

Out\[3\]:

5 rows × 95 columns

In \[4\]:

data.shape

Out\[4\]:

(61878, 95)

In \[5\]:

data.describe()

Out\[5\]:

<table> 
 <thead> 
  <tr> 
   <th></th> 
   <th>id</th> 
   <th>feat_1</th> 
   <th>feat_2</th> 
   <th>feat_3</th> 
   <th>feat_4</th> 
   <th>feat_5</th> 
   <th>feat_6</th> 
   <th>feat_7</th> 
   <th>feat_8</th> 
   <th>feat_9</th> 
   <th>...</th> 
   <th>feat_84</th> 
   <th>feat_85</th> 
   <th>feat_86</th> 
   <th>feat_87</th> 
   <th>feat_88</th> 
   <th>feat_89</th> 
   <th>feat_90</th> 
   <th>feat_91</th> 
   <th>feat_92</th> 
   <th>feat_93</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <th>count</th> 
   <td>61878.000000</td> 
   <td>61878.00000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>...</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
   <td>61878.000000</td> 
  </tr> 
  <tr> 
   <th>mean</th> 
   <td>30939.500000</td> 
   <td>0.38668</td> 
   <td>0.263066</td> 
   <td>0.901467</td> 
   <td>0.779081</td> 
   <td>0.071043</td> 
   <td>0.025696</td> 
   <td>0.193704</td> 
   <td>0.662433</td> 
   <td>1.011296</td> 
   <td>...</td> 
   <td>0.070752</td> 
   <td>0.532306</td> 
   <td>1.128576</td> 
   <td>0.393549</td> 
   <td>0.874915</td> 
   <td>0.457772</td> 
   <td>0.812421</td> 
   <td>0.264941</td> 
   <td>0.380119</td> 
   <td>0.126135</td> 
  </tr> 
  <tr> 
   <th>std</th> 
   <td>17862.784315</td> 
   <td>1.52533</td> 
   <td>1.252073</td> 
   <td>2.934818</td> 
   <td>2.788005</td> 
   <td>0.438902</td> 
   <td>0.215333</td> 
   <td>1.030102</td> 
   <td>2.255770</td> 
   <td>3.474822</td> 
   <td>...</td> 
   <td>1.151460</td> 
   <td>1.900438</td> 
   <td>2.681554</td> 
   <td>1.575455</td> 
   <td>2.115466</td> 
   <td>1.527385</td> 
   <td>4.597804</td> 
   <td>2.045646</td> 
   <td>0.982385</td> 
   <td>1.201720</td> 
  </tr> 
  <tr> 
   <th>min</th> 
   <td>1.000000</td> 
   <td>0.00000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>...</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
  </tr> 
  <tr> 
   <th>25%</th> 
   <td>15470.250000</td> 
   <td>0.00000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>...</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
  </tr> 
  <tr> 
   <th>50%</th> 
   <td>30939.500000</td> 
   <td>0.00000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>...</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
  </tr> 
  <tr> 
   <th>75%</th> 
   <td>46408.750000</td> 
   <td>0.00000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>1.000000</td> 
   <td>0.000000</td> 
   <td>...</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>1.000000</td> 
   <td>0.000000</td> 
   <td>1.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
  </tr> 
  <tr> 
   <th>max</th> 
   <td>61878.000000</td> 
   <td>61.00000</td> 
   <td>51.000000</td> 
   <td>64.000000</td> 
   <td>70.000000</td> 
   <td>19.000000</td> 
   <td>10.000000</td> 
   <td>38.000000</td> 
   <td>76.000000</td> 
   <td>43.000000</td> 
   <td>...</td> 
   <td>76.000000</td> 
   <td>55.000000</td> 
   <td>65.000000</td> 
   <td>67.000000</td> 
   <td>30.000000</td> 
   <td>61.000000</td> 
   <td>130.000000</td> 
   <td>52.000000</td> 
   <td>19.000000</td> 
   <td>87.000000</td> 
  </tr> 
 </tbody> 
</table>

8 rows × 94 columns

In \[7\]:

\# 图形可视化,查看数据分布  
import seaborn as sns

sns.countplot(data.target)

plt.show()

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 12][]

由上图可以看出,该数据类别不均衡,所以需要后期处理

# 数据基本处理 #

数据已经经过脱敏,不再需要特殊处理

## 截取部分数据 ##

In \[8\]:

new1\_data = data\[:10000\]  
new1\_data.shape

Out\[8\]:

(10000, 95)

In \[9\]:

\# 图形可视化,查看数据分布  
import seaborn as sns

sns.countplot(new1\_data.target)

plt.show()

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 13][]

使用上面方式获取数据不可行,然后使用随机欠采样获取响应的数据

In \[10\]:

\# 随机欠采样获取数据  
\# 首先需要确定特征值\\标签值

y = data\["target"\]  
x = data.drop(\["id", "target"\], axis=1)

In \[11\]:

x.head()

Out\[11\]:

5 rows × 93 columns

In \[12\]:

y.head()

Out\[12\]:

0    Class_1
    1    Class_1
    2    Class_1
    3    Class_1
    4    Class_1
    Name: target, dtype: object

In \[14\]:

\# 欠采样获取数据  
from imblearn.under\_sampling import RandomUnderSampler

rus = RandomUnderSampler(random\_state=0)

X\_resampled, y\_resampled = rus.fit\_resample(x, y)

In \[15\]:

x.shape, y.shape

Out\[15\]:

((61878, 93), (61878,))

In \[16\]:

X\_resampled.shape, y\_resampled.shape

Out\[16\]:

((17361, 93), (17361,))

In \[17\]:

\# 图形可视化,查看数据分布  
import seaborn as sns

sns.countplot(y\_resampled)

plt.show()

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 14][]

## 把标签值转换为数字 ##

In \[18\]:

y\_resampled.head()

Out\[18\]:

0    Class_1
    1    Class_1
    2    Class_1
    3    Class_1
    4    Class_1
    Name: target, dtype: object

In \[19\]:

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()  
y\_resampled = le.fit\_transform(y\_resampled)

In \[20\]:

y\_resampled

Out\[20\]:

array([0, 0, 0, ..., 8, 8, 8])

## 分割数据 ##

In \[21\]:

from sklearn.model\_selection import train\_test\_split

x\_train, x\_test, y\_train, y\_test = train\_test\_split(X\_resampled, y\_resampled, test\_size=0.2)

In \[23\]:

x\_train.shape, y\_train.shape

Out\[23\]:

((13888, 93), (13888,))

In \[24\]:

x\_test.shape, y\_test.shape

Out\[24\]:

((3473, 93), (3473,))

# 模型训练 #

## 基本模型训练 ##

In \[26\]:

from sklearn.ensemble import RandomForestClassifier  
rf = RandomForestClassifier(oob\_score=True)  
rf.fit(x\_train, y\_train)

Out\[26\]:

RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
                           criterion='gini', max_depth=None, max_features='auto',
                           max_leaf_nodes=None, max_samples=None,
                           min_impurity_decrease=0.0, min_impurity_split=None,
                           min_samples_leaf=1, min_samples_split=2,
                           min_weight_fraction_leaf=0.0, n_estimators=100,
                           n_jobs=None, oob_score=True, random_state=None,
                           verbose=0, warm_start=False)

In \[28\]:

y\_pre = rf.predict(x\_test)  
y\_pre

Out\[28\]:

array([7, 7, 0, ..., 1, 1, 0])

In \[29\]:

rf.score(x\_test, y\_test)

Out\[29\]:

0.7782896631154621

In \[30\]:

rf.oob\_score\_

Out\[30\]:

0.7659850230414746

In \[31\]:

\# 图形可视化,查看数据分布

import seaborn as sns

sns.countplot(y_pre)

plt.show()

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 13][]

In \[33\]:

\# logloss模型评估  
from sklearn.metrics import log\_loss

\#log\_loss(y\_test, y\_pre, eps=1e-15, normalize=True)

In \[35\]:

y\_test, y\_pre

Out\[35\]:

(array([7, 7, 8, ..., 1, 1, 0]), array([7, 7, 0, ..., 1, 1, 0]))

上面报错原因:logloss使用过程中,必须要求将输出用one-hot表示,

需要将这个多类别问题的输出结果通过OneHotEncoder修改为如下:

In \[37\]:

from sklearn.preprocessing import OneHotEncoder

one\_hot = OneHotEncoder(sparse=False)

y\_test1 = one\_hot.fit\_transform(y\_test.reshape(-1, 1))  
y\_pre1 = one\_hot.fit\_transform(y\_pre.reshape(-1, 1))

In \[38\]:

y\_test1

Out\[38\]:

array([[0., 0., 0., ..., 0., 1., 0.],
           [0., 0., 0., ..., 0., 1., 0.],
           [0., 0., 0., ..., 0., 0., 1.],
           ...,
           [0., 1., 0., ..., 0., 0., 0.],
           [0., 1., 0., ..., 0., 0., 0.],
           [1., 0., 0., ..., 0., 0., 0.]])

In \[39\]:

y\_pre1

Out\[39\]:

array([[0., 0., 0., ..., 0., 1., 0.],
           [0., 0., 0., ..., 0., 1., 0.],
           [1., 0., 0., ..., 0., 0., 0.],
           ...,
           [0., 1., 0., ..., 0., 0., 0.],
           [0., 1., 0., ..., 0., 0., 0.],
           [1., 0., 0., ..., 0., 0., 0.]])

In \[40\]:

\# logloss模型评估

log_loss(y_test1, y_pre1, eps=1e-15, normalize=True)

Out\[40\]:

7.65760375009538

In \[41\]:

\# 改变预测值的输出模式,让输出结果为百分占比,降低logloss值

y_pre_proba = rf.predict_proba(x_test)

In \[42\]:

y\_pre\_proba

Out\[42\]:

array([[0.04, 0.  , 0.  , ..., 0.01, 0.92, 0.01],
           [0.03, 0.02, 0.03, ..., 0.2 , 0.48, 0.08],
           [0.41, 0.06, 0.06, ..., 0.05, 0.04, 0.31],
           ...,
           [0.  , 0.49, 0.28, ..., 0.02, 0.  , 0.01],
           [0.01, 0.54, 0.09, ..., 0.  , 0.  , 0.02],
           [0.31, 0.03, 0.02, ..., 0.11, 0.13, 0.13]])

In \[43\]:

rf.oob\_score\_

Out\[43\]:

0.7659850230414746

In \[44\]:

\# logloss模型评估

log_loss(y_test1, y_pre_proba, eps=1e-15, normalize=True)

Out\[44\]:

0.747188586960504

## 模型调优 ##

n\_estimators, max\_feature, max\_depth, min\_samples\_leaf

### 确定最优的n\_estimators ###

In \[45\]:

\# 确定n\_estimators的取值范围  
tuned\_parameters = range(10, 200, 10)

\# 创建添加accuracy的一个numpy  
accuracy\_t = np.zeros(len(tuned\_parameters))

\# 创建添加error的一个numpy  
error\_t = np.zeros(len(tuned\_parameters))

\# 调优过程实现  
for j, one\_parameter in enumerate(tuned\_parameters):  
    rf2 = RandomForestClassifier(n\_estimators=one\_parameter,   
                                 max\_depth=10,   
                                 max\_features=10,   
                                 min\_samples\_leaf=10,   
                                 oob\_score=True,   
                                 random\_state=0,   
                                 n\_jobs=-1)  
      
    rf2.fit(x\_train, y\_train)  
      
    \# 输出accuracy  
    accuracy\_t\[j\] = rf2.oob\_score\_  
      
    \# 输出log\_loss  
    y\_pre = rf2.predict\_proba(x\_test)  
    error\_t\[j\] = log\_loss(y\_test, y\_pre, eps=1e-15, normalize=True)  
      
    print(error\_t)

[1.1363637 0.        0.        0.        0.        0.        0.
     0.        0.        0.        0.        0.        0.        0.
     0.        0.        0.        0.        0.       ]

[1.1363637  1.12502792 0.         0.         0.         0.
     0.         0.         0.         0.         0.         0.
     0.         0.         0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 0.         0.         0.
     0.         0.         0.         0.         0.         0.
     0.         0.         0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 0.         0.
     0.         0.         0.         0.         0.         0.
     0.         0.         0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 0.
     0.         0.         0.         0.         0.         0.
     0.         0.         0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     0.         0.         0.         0.         0.         0.
     0.         0.         0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 0.         0.         0.         0.         0.
     0.         0.         0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 1.1209923  0.         0.         0.         0.
     0.         0.         0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 1.1209923  1.12231238 0.         0.         0.
     0.         0.         0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 1.1209923  1.12231238 1.11887248 0.         0.
     0.         0.         0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 1.1209923  1.12231238 1.11887248 1.11682286 0.
     0.         0.         0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 1.1209923  1.12231238 1.11887248 1.11682286 1.11668191
     0.         0.         0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 1.1209923  1.12231238 1.11887248 1.11682286 1.11668191
     1.11607429 0.         0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 1.1209923  1.12231238 1.11887248 1.11682286 1.11668191
     1.11607429 1.11658966 0.         0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 1.1209923  1.12231238 1.11887248 1.11682286 1.11668191
     1.11607429 1.11658966 1.11551087 0.         0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 1.1209923  1.12231238 1.11887248 1.11682286 1.11668191
     1.11607429 1.11658966 1.11551087 1.11454353 0.         0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 1.1209923  1.12231238 1.11887248 1.11682286 1.11668191
     1.11607429 1.11658966 1.11551087 1.11454353 1.11539224 0.
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 1.1209923  1.12231238 1.11887248 1.11682286 1.11668191
     1.11607429 1.11658966 1.11551087 1.11454353 1.11539224 1.11527522
     0.        ]
    [1.1363637  1.12502792 1.12888587 1.12663772 1.12185466 1.11919361
     1.11933586 1.1209923  1.12231238 1.11887248 1.11682286 1.11668191
     1.11607429 1.11658966 1.11551087 1.11454353 1.11539224 1.11527522
     1.11545188]

In \[50\]:

\# 优化结果过程可视化  
fig,axes = plt.subplots(nrows=1, ncols=2, figsize=(20, 4), dpi=100)

axes\[0\].plot(tuned\_parameters, error\_t)  
axes\[1\].plot(tuned\_parameters, accuracy\_t)

axes\[0\].set\_xlabel("n\_estimators")  
axes\[0\].set\_ylabel("error\_t")  
axes\[1\].set\_xlabel("n\_estimators")  
axes\[1\].set\_ylabel("accuracy\_t")

axes\[0\].grid(True)  
axes\[1\].grid(True)

plt.show()

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 15][]

经过图像展示,最后确定n\_estimators=175的时候,表现效果不错

### 确定最优的max\_features ###

In \[51\]:

\# 确定n\_estimators的取值范围  
tuned\_parameters = range(5, 40, 5)

\# 创建添加accuracy的一个numpy  
accuracy\_t = np.zeros(len(tuned\_parameters))

\# 创建添加error的一个numpy  
error\_t = np.zeros(len(tuned\_parameters))

\# 调优过程实现  
for j, one\_parameter in enumerate(tuned\_parameters):  
    rf2 = RandomForestClassifier(n\_estimators=175,   
                                 max\_depth=10,   
                                 max\_features=one\_parameter,   
                                 min\_samples\_leaf=10,   
                                 oob\_score=True,   
                                 random\_state=0,   
                                 n\_jobs=-1)  
      
    rf2.fit(x\_train, y\_train)  
      
    \# 输出accuracy  
    accuracy\_t\[j\] = rf2.oob\_score\_  
      
    \# 输出log\_loss  
    y\_pre = rf2.predict\_proba(x\_test)  
    error\_t\[j\] = log\_loss(y\_test, y\_pre, eps=1e-15, normalize=True)  
      
    print(error\_t)

[1.21048299 0.         0.         0.         0.         0.
     0.        ]
    [1.21048299 1.11570883 0.         0.         0.         0.
     0.        ]
    [1.21048299 1.11570883 1.08477789 0.         0.         0.
     0.        ]
    [1.21048299 1.11570883 1.08477789 1.06988755 0.         0.
     0.        ]
    [1.21048299 1.11570883 1.08477789 1.06988755 1.06604809 0.
     0.        ]
    [1.21048299 1.11570883 1.08477789 1.06988755 1.06604809 1.05863125
     0.        ]
    [1.21048299 1.11570883 1.08477789 1.06988755 1.06604809 1.05863125
     1.06050897]

In \[52\]:

\# 优化结果过程可视化  
fig,axes = plt.subplots(nrows=1, ncols=2, figsize=(20, 4), dpi=100)

axes\[0\].plot(tuned\_parameters, error\_t)  
axes\[1\].plot(tuned\_parameters, accuracy\_t)

axes\[0\].set\_xlabel("max\_features")  
axes\[0\].set\_ylabel("error\_t")  
axes\[1\].set\_xlabel("max\_features")  
axes\[1\].set\_ylabel("accuracy\_t")

axes\[0\].grid(True)  
axes\[1\].grid(True)

plt.show()

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 16][]

经过图像展示,最后确定max\_features=15的时候,表现效果不错

### 确定最优的max\_depth ###

In \[53\]:

\# 确定n\_estimators的取值范围  
tuned\_parameters = range(10, 100, 10)

\# 创建添加accuracy的一个numpy  
accuracy\_t = np.zeros(len(tuned\_parameters))

\# 创建添加error的一个numpy  
error\_t = np.zeros(len(tuned\_parameters))

\# 调优过程实现  
for j, one\_parameter in enumerate(tuned\_parameters):  
    rf2 = RandomForestClassifier(n\_estimators=175,   
                                 max\_depth=one\_parameter,   
                                 max\_features=15,   
                                 min\_samples\_leaf=10,   
                                 oob\_score=True,   
                                 random\_state=0,   
                                 n\_jobs=-1)  
      
    rf2.fit(x\_train, y\_train)  
      
    \# 输出accuracy  
    accuracy\_t\[j\] = rf2.oob\_score\_  
      
    \# 输出log\_loss  
    y\_pre = rf2.predict\_proba(x\_test)  
    error\_t\[j\] = log\_loss(y\_test, y\_pre, eps=1e-15, normalize=True)  
      
    print(error\_t)

[1.08477789 0.         0.         0.         0.         0.
     0.         0.         0.        ]
    [1.08477789 0.85395849 0.         0.         0.         0.
     0.         0.         0.        ]
    [1.08477789 0.85395849 0.82965202 0.         0.         0.
     0.         0.         0.        ]
    [1.08477789 0.85395849 0.82965202 0.83011988 0.         0.
     0.         0.         0.        ]
    [1.08477789 0.85395849 0.82965202 0.83011988 0.8302702  0.
     0.         0.         0.        ]
    [1.08477789 0.85395849 0.82965202 0.83011988 0.8302702  0.8302702
     0.         0.         0.        ]
    [1.08477789 0.85395849 0.82965202 0.83011988 0.8302702  0.8302702
     0.8302702  0.         0.        ]
    [1.08477789 0.85395849 0.82965202 0.83011988 0.8302702  0.8302702
     0.8302702  0.8302702  0.        ]
    [1.08477789 0.85395849 0.82965202 0.83011988 0.8302702  0.8302702
     0.8302702  0.8302702  0.8302702 ]

In \[54\]:

\# 优化结果过程可视化  
fig,axes = plt.subplots(nrows=1, ncols=2, figsize=(20, 4), dpi=100)

axes\[0\].plot(tuned\_parameters, error\_t)  
axes\[1\].plot(tuned\_parameters, accuracy\_t)

axes\[0\].set\_xlabel("max\_depth")  
axes\[0\].set\_ylabel("error\_t")  
axes\[1\].set\_xlabel("max\_depth")  
axes\[1\].set\_ylabel("accuracy\_t")

axes\[0\].grid(True)  
axes\[1\].grid(True)

plt.show()

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 17][]

经过图像展示,最后确定max\_depth=30的时候,表现效果不错

### 确定最优的min\_sample\_leaf ###

In \[56\]:

\# 确定n\_estimators的取值范围  
tuned\_parameters = range(1, 10, 2)

\# 创建添加accuracy的一个numpy  
accuracy\_t = np.zeros(len(tuned\_parameters))

\# 创建添加error的一个numpy  
error\_t = np.zeros(len(tuned\_parameters))

\# 调优过程实现  
for j, one\_parameter in enumerate(tuned\_parameters):  
    rf2 = RandomForestClassifier(n\_estimators=175,   
                                 max\_depth=30,   
                                 max\_features=15,   
                                 min\_samples\_leaf=one\_parameter,   
                                 oob\_score=True,   
                                 random\_state=0,   
                                 n\_jobs=-1)  
      
    rf2.fit(x\_train, y\_train)  
      
    \# 输出accuracy  
    accuracy\_t\[j\] = rf2.oob\_score\_  
      
    \# 输出log\_loss  
    y\_pre = rf2.predict\_proba(x\_test)  
    error\_t\[j\] = log\_loss(y\_test, y\_pre, eps=1e-15, normalize=True)  
      
    print(error\_t)

[0.71382973 0.         0.         0.         0.        ]
    [0.71382973 0.74611594 0.         0.         0.        ]
    [0.71382973 0.74611594 0.77748299 0.         0.        ]
    [0.71382973 0.74611594 0.77748299 0.80311457 0.        ]
    [0.71382973 0.74611594 0.77748299 0.80311457 0.82513751]

In \[57\]:

\# 优化结果过程可视化  
fig,axes = plt.subplots(nrows=1, ncols=2, figsize=(20, 4), dpi=100)

axes\[0\].plot(tuned\_parameters, error\_t)  
axes\[1\].plot(tuned\_parameters, accuracy\_t)

axes\[0\].set\_xlabel("min\_sample\_leaf")  
axes\[0\].set\_ylabel("error\_t")  
axes\[1\].set\_xlabel("min\_sample\_leaf")  
axes\[1\].set\_ylabel("accuracy\_t")

axes\[0\].grid(True)  
axes\[1\].grid(True)

plt.show()

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 18][]

经过图像展示,最后确定min\_sample\_leaf=1的时候,表现效果不错

## 确定最优模型 ##

n\_estimators=175,

max\_depth=30,

max\_features=15,

min\_samples\_leaf=1,

In \[59\]:

rf3 = RandomForestClassifier(n\_estimators=175, max\_depth=30, max\_features=15, min\_samples\_leaf=1,   
                             oob\_score=True, random\_state=40, n\_jobs=-1)

rf3.fit(x\_train, y\_train)

rf3.fit(x_train, y_train)

Out\[59\]:

RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
                           criterion='gini', max_depth=30, max_features=15,
                           max_leaf_nodes=None, max_samples=None,
                           min_impurity_decrease=0.0, min_impurity_split=None,
                           min_samples_leaf=1, min_samples_split=2,
                           min_weight_fraction_leaf=0.0, n_estimators=175,
                           n_jobs=-1, oob_score=True, random_state=40, verbose=0,
                           warm_start=False)

In \[60\]:

rf3.score(x\_test, y\_test)

Out\[60\]:

0.7837604376619637

In \[61\]:

rf3.oob\_score\_

Out\[61\]:

0.7746255760368663

In \[62\]:

y\_pre\_proba1 = rf3.predict\_proba(x\_test)

log\_loss(y\_test, y\_pre\_proba1)

Out\[62\]:

0.7113183040048942

# 生成提交数据[¶][Link 2] #

In \[63\]:

test\_data = pd.read\_csv("./data/otto/test.csv")

In \[64\]:

test\_data.head()

Out\[64\]:

5 rows × 94 columns

In \[65\]:

test\_data\_drop\_id = test\_data.drop(\["id"\], axis=1)  
test\_data\_drop\_id.head()

Out\[65\]:

5 rows × 93 columns

In \[66\]:

y\_pre\_test = rf3.predict\_proba(test\_data\_drop\_id)

In \[67\]:

y\_pre\_test

Out\[67\]:

array([[3.42857143e-02, 4.00000000e-02, 6.00000000e-02, ...,
            4.01785714e-02, 5.71428571e-03, 2.28571429e-02],
           [1.14285714e-01, 4.00000000e-02, 3.42857143e-02, ...,
            2.85714286e-02, 2.17142857e-01, 2.85714286e-02],
           [5.71428571e-03, 0.00000000e+00, 0.00000000e+00, ...,
            0.00000000e+00, 5.71428571e-03, 0.00000000e+00],
           ...,
           [1.72644377e-02, 2.48465399e-01, 4.16172412e-01, ...,
            3.42857143e-02, 1.21580547e-04, 5.71428571e-03],
           [1.72287863e-02, 3.06668815e-01, 1.07797351e-01, ...,
            3.93841747e-04, 0.00000000e+00, 5.71428571e-03],
           [1.92414742e-02, 2.14828481e-01, 3.61958916e-01, ...,
            1.60309793e-01, 0.00000000e+00, 1.63868444e-03]])

In \[68\]:

result\_data = pd.DataFrame(y\_pre\_test, columns=\["Class\_"+str(i) for i in range(1, 10)\])

In \[69\]:

result\_data.head()

Out\[69\]:

<table> 
 <thead> 
  <tr> 
   <th></th> 
   <th>Class_1</th> 
   <th>Class_2</th> 
   <th>Class_3</th> 
   <th>Class_4</th> 
   <th>Class_5</th> 
   <th>Class_6</th> 
   <th>Class_7</th> 
   <th>Class_8</th> 
   <th>Class_9</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <th>0</th> 
   <td>0.034286</td> 
   <td>0.040000</td> 
   <td>0.060000</td> 
   <td>0.774107</td> 
   <td>0.000000</td> 
   <td>0.022857</td> 
   <td>0.040179</td> 
   <td>0.005714</td> 
   <td>0.022857</td> 
  </tr> 
  <tr> 
   <th>1</th> 
   <td>0.114286</td> 
   <td>0.040000</td> 
   <td>0.034286</td> 
   <td>0.034286</td> 
   <td>0.017143</td> 
   <td>0.485714</td> 
   <td>0.028571</td> 
   <td>0.217143</td> 
   <td>0.028571</td> 
  </tr> 
  <tr> 
   <th>2</th> 
   <td>0.005714</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.988571</td> 
   <td>0.000000</td> 
   <td>0.005714</td> 
   <td>0.000000</td> 
  </tr> 
  <tr> 
   <th>3</th> 
   <td>0.027714</td> 
   <td>0.215714</td> 
   <td>0.300000</td> 
   <td>0.277143</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.017143</td> 
   <td>0.006286</td> 
   <td>0.156000</td> 
  </tr> 
  <tr> 
   <th>4</th> 
   <td>0.237060</td> 
   <td>0.000080</td> 
   <td>0.005834</td> 
   <td>0.000000</td> 
   <td>0.005954</td> 
   <td>0.023057</td> 
   <td>0.018190</td> 
   <td>0.199658</td> 
   <td>0.510166</td> 
  </tr> 
 </tbody> 
</table>

In \[70\]:

result\_data.insert(loc=0, column="id", value=test\_data.id)

In \[71\]:

result\_data.head()

Out\[71\]:

<table> 
 <thead> 
  <tr> 
   <th></th> 
   <th>id</th> 
   <th>Class_1</th> 
   <th>Class_2</th> 
   <th>Class_3</th> 
   <th>Class_4</th> 
   <th>Class_5</th> 
   <th>Class_6</th> 
   <th>Class_7</th> 
   <th>Class_8</th> 
   <th>Class_9</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <th>0</th> 
   <td>1</td> 
   <td>0.034286</td> 
   <td>0.040000</td> 
   <td>0.060000</td> 
   <td>0.774107</td> 
   <td>0.000000</td> 
   <td>0.022857</td> 
   <td>0.040179</td> 
   <td>0.005714</td> 
   <td>0.022857</td> 
  </tr> 
  <tr> 
   <th>1</th> 
   <td>2</td> 
   <td>0.114286</td> 
   <td>0.040000</td> 
   <td>0.034286</td> 
   <td>0.034286</td> 
   <td>0.017143</td> 
   <td>0.485714</td> 
   <td>0.028571</td> 
   <td>0.217143</td> 
   <td>0.028571</td> 
  </tr> 
  <tr> 
   <th>2</th> 
   <td>3</td> 
   <td>0.005714</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.988571</td> 
   <td>0.000000</td> 
   <td>0.005714</td> 
   <td>0.000000</td> 
  </tr> 
  <tr> 
   <th>3</th> 
   <td>4</td> 
   <td>0.027714</td> 
   <td>0.215714</td> 
   <td>0.300000</td> 
   <td>0.277143</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.017143</td> 
   <td>0.006286</td> 
   <td>0.156000</td> 
  </tr> 
  <tr> 
   <th>4</th> 
   <td>5</td> 
   <td>0.237060</td> 
   <td>0.000080</td> 
   <td>0.005834</td> 
   <td>0.000000</td> 
   <td>0.005954</td> 
   <td>0.023057</td> 
   <td>0.018190</td> 
   <td>0.199658</td> 
   <td>0.510166</td> 
  </tr> 
 </tbody> 
</table>

In \[72\]:

result\_data.to\_csv("./data/otto/submission.csv", index=False)

--------------------

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 19][]

--------------------

# 5.4 Boosting #

## 1 什么是boosting ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 20][]

**随着学习的积累从弱到强**

**简而言之：每新加入一个弱学习器，整体能力就会得到提升**

代表算法：Adaboost，GBDT，XGBoost，LightGBM

## 2 实现过程： ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 21][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 22][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 23][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 24][]

## 3 bagging集成与boosting集成的区别： ##

*  区别一:数据方面
    
     *  Bagging：对数据进行采样训练；
     *  Boosting：根据前一轮学习结果调整数据的重要性。
 *  区别二:投票方面
    
     *  Bagging：所有学习器平权投票；
     *  Boosting：对学习器进行加权投票。
 *  区别三:学习顺序
    
     *  Bagging的学习是并行的，每个学习器没有依赖关系；
     *  Boosting学习是串行，学习有先后顺序。
 *  区别四:主要作用
    
     *  Bagging主要用于提高泛化性能（解决过拟合，也可以说降低方差）
     *  Boosting主要用于提高训练精度 （解决欠拟合，也可以说降低偏差）

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 25][]

## 4 AdaBoost介绍 ##

### 4.1 构造过程细节 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 26][]

![20200227091136483.png][]

### 4.2 关键点剖析 ###

**如何确认投票权重？**

**如何调整数据分布？**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 27][]

### 4.3 案例： ###

给定下面这张训练数据表所示的数据，假设弱分类器由xv产生，其阈值v使该分类器在训练数据集上的分类误差率最低，试用Adaboost算法学习一个强分类器。

![20200227091154422.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 28][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 29][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 30][]

![20200227092408223.png][]![20200227092417368.png][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 31][]

### 4.4 api介绍 ###

*  from sklearn.ensemble import AdaBoostClassifier
    
     *  [api链接:][api][https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html\#sklearn.ensemble.AdaBoostClassifier][api]

## 5 小结 ##

*  什么是Boosting 【知道】
    
     *  随着学习的积累从弱到强
     *  代表算法：Adaboost，GBDT，XGBoost，LightGBM
 *  bagging和boosting的区别【知道】
    
     *  区别一:数据方面
        
         *  Bagging：对数据进行采样训练；
         *  Boosting：根据前一轮学习结果调整数据的重要性。
     *  区别二:投票方面
        
         *  Bagging：所有学习器平权投票；
         *  Boosting：对学习器进行加权投票。
     *  区别三:学习顺序
        
         *  Bagging的学习是并行的，每个学习器没有依赖关系；
         *  Boosting学习是串行，学习有先后顺序。
     *  区别四:主要作用
        
         *  Bagging主要用于提高泛化性能（解决过拟合，也可以说降低方差）
         *  Boosting主要用于提高训练精度 （解决欠拟合，也可以说降低偏差）
 *  AdaBoost构造过程【知道】
    
     *  步骤一：初始化训练数据权重相等，训练第一个学习器;
     *  步骤二：AdaBoost反复学习基本分类器;
     *  步骤三：对m个学习器进行加权投票

--------------------

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 32][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 33][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 34][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 35][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 36][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 37][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 38][]

--------------------

# 5.5 GBDT介绍 #

GBDT 的全称是 Gradient Boosting Decision Tree，梯度提升树，在传统机器学习算法中，GBDT算的上TOP3的算法。想要理解GBDT的真正意义，那就必须理解GBDT中的**Gradient Boosting** 和**Decision Tree**分别是什么？

## 1 Decision Tree：CART回归树 ##

首先，**GBDT使用的决策树是CART回归树**，无论是处理回归问题还是二分类以及多分类，GBDT使用的决策树通通都是都是CART回归树。

*  为什么不用CART分类树呢？
    
     *  因为**GBDT每次迭代要拟合的是梯度值**，是连续值所以要用回归树。

对于回归树算法来说最重要的是寻找最佳的划分点，那么回归树中的可划分点包含了所有特征的所有可取的值。

在分类树中最佳划分点的判别标准是熵或者基尼系数，都是用纯度来衡量的，但是在回归树中的样本标签是连续数值，所以再使用熵之类的指标不再合适，**取而代之的是平方误差，它能很好的评判拟合程度。**

### 1.1 回归树生成算法（复习） ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 39][]

## 2 Gradient Boosting： 拟合负梯度 ##

梯度提升树（Grandient Boosting）是提升树（Boosting Tree）的一种改进算法，所以在讲梯度提升树之前先来说一下提升树。

先来个通俗理解：假如有个人30岁，我们首先用20岁去拟合，发现损失有10岁，这时我们用6岁去拟合剩下的损失，发现差距还有4岁，第三轮我们用3岁拟合剩下的差距，差距就只有一岁了。如果我们的迭代轮数还没有完，可以继续迭代下面，每一轮迭代，拟合的岁数误差都会减小。最后将每次拟合的岁数加起来便是模型输出的结果。

**提升树算法：**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 40][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 41][]

当损失函数是平方损失和指数损失函数时，梯度提升树每一步优化是很简单的，但是对于一般损失函数而言，往往每一步优化起来不那么容易。

针对这一问题，Friedman提出了梯度提升树算法，这是利用最速下降的近似方法，其关**键是利用损失函数的负梯度作为提升树算法中的残差的近似值**。

那么负梯度长什么样呢？

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 42][]

此时我们发现GBDT的负梯度就是残差，所以说对于回归问题，我们要拟合的就是残差。

那么对于分类问题呢？

*  二分类和多分类的损失函数都是logloss。

本文以回归问题为例进行讲解。

## 3 GBDT算法原理 ##

上面两节分别将Decision Tree和Gradient Boosting介绍完了，下面将这两部分组合在一起就是我们的GBDT了。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 43][]

![20200227092010122.png][]

## 4 实例介绍 ##

### 4.1 数据介绍 ###

根据如下数据，预测最后一个样本的身高。

### 4.2 模型训练 ###

**4.2.1 设置参数：**

*  学习率：learning\_rate=0.1
 *  迭代次数：n\_trees=5
 *  树的深度：max\_depth=3

4.2.2 开始训练

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 44][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 45][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 46][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 47][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 48][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 49][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 50][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 51][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 52][]

![20200227092234605.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 53][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 54][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 55][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 56][]

--------------------

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 57][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 58][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 59][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 60][]

1.年龄x=5 7为左节点SEl，年龄x=21 30为右节点SEr。
    2.年龄x 左节点SEl：
    	负梯度(残差)作为标签值y，然后计算左节点的标签值y的平均值c1=(-0.375+(-0.175))/2=-0.275
    	求平方差(标签值y-c1)^2：SEl=[-0.375-(-0.275)]^2 + [-0.175-(-0.275)]^2 = 0.02
    3.年龄x 右节点SEr：
    	负梯度(残差)作为标签值y，然后计算右节点的标签值y的平均值c2=(0.225+0.325)/2=0.275
    	求平方差(标签值y-c2)^2：SEr=[0.225-0.275]^2 + [0.325-0.275]^2 = 0.005

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 61][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 62][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 63][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 64][]

[20191009191333910.png]: https://img-blog.csdnimg.cn/20191009191333910.png
[Link 1]: http://www.rimengshe.com/
[AI_Keras PyTorch MXNet TensorFlow PaddlePaddle]: https://blog.csdn.net/zimiao552147572/article/details/88867161
[Bagging_Boosting_GBDT]: https://blog.csdn.net/zimiao552147572/article/details/104529615
[XGBoost]: https://blog.csdn.net/zimiao552147572/article/details/104658658
[lightGBM]: https://blog.csdn.net/zimiao552147572/article/details/104659336
[lightGBM 1]: https://blog.csdn.net/zimiao552147572/article/details/104659386
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200227085501358.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200227085519439.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200227085625554.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200227085619739.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200227085612598.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/2020022708563735.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200227085645509.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200227085654380.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200227085725617.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[20200227085732837.png]: https://img-blog.csdnimg.cn/20200227085732837.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200227085759550.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[https_www.kaggle.com_c_otto-group-product-classification-challenge_overview]: https://www.kaggle.com/c/otto-group-product-classification-challenge/overview
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20200227085927567.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[20200227085945233.png]: https://img-blog.csdnimg.cn/20200227085945233.png
[20200227090006970.png]: https://img-blog.csdnimg.cn/20200227090006970.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200227090007578.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/2020022709003177.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20200227090031331.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20200227090031352.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/20200227090031770.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 16]: https://img-blog.csdnimg.cn/20200227090031104.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 17]: https://img-blog.csdnimg.cn/20200227090031165.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 18]: https://img-blog.csdnimg.cn/20200227090031407.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[Link 2]: http://localhost:8888/notebooks/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/02_%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0%E7%AE%97%E6%B3%95day08/4.RF_otto.ipynb#%E7%94%9F%E6%88%90%E6%8F%90%E4%BA%A4%E6%95%B0%E6%8D%AE
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 19]: https://img-blog.csdnimg.cn/20200227091649326.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 20]: https://img-blog.csdnimg.cn/2020022709095559.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 21]: https://img-blog.csdnimg.cn/20200227091013186.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 22]: https://img-blog.csdnimg.cn/20200227091020756.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 23]: https://img-blog.csdnimg.cn/20200227091027888.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 24]: https://img-blog.csdnimg.cn/20200227091033460.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 25]: https://img-blog.csdnimg.cn/20200227091059751.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 26]: https://img-blog.csdnimg.cn/20200227091128292.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[20200227091136483.png]: https://img-blog.csdnimg.cn/20200227091136483.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 27]: https://img-blog.csdnimg.cn/20200227091145176.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[20200227091154422.png]: https://img-blog.csdnimg.cn/20200227091154422.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 28]: https://img-blog.csdnimg.cn/20200227091203834.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 29]: https://img-blog.csdnimg.cn/20200227091212961.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 30]: https://img-blog.csdnimg.cn/20200227091225739.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[20200227092408223.png]: https://img-blog.csdnimg.cn/20200227092408223.png
[20200227092417368.png]: https://img-blog.csdnimg.cn/20200227092417368.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 31]: https://img-blog.csdnimg.cn/20200227092434266.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[api]: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html#sklearn.ensemble.AdaBoostClassifier
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 32]: https://img-blog.csdnimg.cn/20200227091718686.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 33]: https://img-blog.csdnimg.cn/20200227091729286.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 34]: https://img-blog.csdnimg.cn/20200227091740430.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 35]: https://img-blog.csdnimg.cn/20200227091750520.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 36]: https://img-blog.csdnimg.cn/20200227091757875.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 37]: https://img-blog.csdnimg.cn/20200227091808205.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 38]: https://img-blog.csdnimg.cn/20200227091813980.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 39]: https://img-blog.csdnimg.cn/20200227091905261.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 40]: https://img-blog.csdnimg.cn/20200227091919412.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 41]: https://img-blog.csdnimg.cn/20200227091925785.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 42]: https://img-blog.csdnimg.cn/20200227091950655.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 43]: https://img-blog.csdnimg.cn/20200227092002708.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[20200227092010122.png]: https://img-blog.csdnimg.cn/20200227092010122.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 44]: https://img-blog.csdnimg.cn/2020022709202729.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 45]: https://img-blog.csdnimg.cn/20200227092039413.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 46]: https://img-blog.csdnimg.cn/20200227092100578.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 47]: https://img-blog.csdnimg.cn/20200227092111322.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 48]: https://img-blog.csdnimg.cn/20200227092127558.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 49]: https://img-blog.csdnimg.cn/20200227092135487.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 50]: https://img-blog.csdnimg.cn/20200227092144617.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 51]: https://img-blog.csdnimg.cn/20200227092159447.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 52]: https://img-blog.csdnimg.cn/20200227092211663.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[20200227092234605.png]: https://img-blog.csdnimg.cn/20200227092234605.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 53]: https://img-blog.csdnimg.cn/20200227092246392.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 54]: https://img-blog.csdnimg.cn/20200227092259692.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 55]: https://img-blog.csdnimg.cn/20200227092315424.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 56]: https://img-blog.csdnimg.cn/2020022709233370.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 57]: https://img-blog.csdnimg.cn/20200227092506838.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 58]: https://img-blog.csdnimg.cn/20200227092517313.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 59]: https://img-blog.csdnimg.cn/20200227092522422.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 60]: https://img-blog.csdnimg.cn/20200227092527103.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 61]: https://img-blog.csdnimg.cn/20200227092537730.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 62]: https://img-blog.csdnimg.cn/20200227092546416.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 63]: https://img-blog.csdnimg.cn/20200227092553451.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg_size_16_color_FFFFFF_t_70 64]: https://img-blog.csdnimg.cn/20200227092559369.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ppbWlhbzU1MjE0NzU3Mg==,size_16,color_FFFFFF,t_70