Pytorch：torch.nn.Parameter理解

怼烎@ 2023-10-02 15:03 38阅读 0赞

Pytorch官网对torch.nn.Parameter()的解释：  
![在这里插入图片描述][watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5b-N6ICF44Gu5Lmx5aSq6YOO_size_20_color_FFFFFF_t_70_g_se_x_16_pic_center]  
torch.nn.Parameter是继承自torch.Tensor的子类，其主要作用是作为nn.Module中的可训练参数使用。它与torch.Tensor的区别就是nn.Parameter会自动被认为是module的可训练参数，即加入到parameter()这个迭代器中去；而module中非nn.Parameter()的普通tensor是不在parameter中的。  
注意到，nn.Parameter的对象的requires\_grad属性的默认值是True，即是可被训练的，这与torth.Tensor对象的默认值相反。  
在nn.Module类中，pytorch也是使用nn.Parameter来对每一个module的参数进行初始化的。以nn.Linear为例：

class Linear(Module):
        r"""Applies a linear transformation to the incoming data: :math:`y = xA^T + b`
        Args:
            in_features: size of each input sample
            out_features: size of each output sample
            bias: If set to ``False``, the layer will not learn an additive bias.
                Default: ``True``
        Shape:
            - Input: :math:`(N, *, H_{in})` where :math:`*` means any number of
              additional dimensions and :math:`H_{in} = \text{in\_features}`
            - Output: :math:`(N, *, H_{out})` where all but the last dimension
              are the same shape as the input and :math:`H_{out} = \text{out\_features}`.
        Attributes:
            weight: the learnable weights of the module of shape
                :math:`(\text{out\_features}, \text{in\_features})`. The values are
                initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})`, where
                :math:`k = \frac{1}{\text{in\_features}}`
            bias:   the learnable bias of the module of shape :math:`(\text{out\_features})`.
                    If :attr:`bias` is ``True``, the values are initialized from
                    :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where
                    :math:`k = \frac{1}{\text{in\_features}}`
        Examples::
            >>> m = nn.Linear(20, 30)
            >>> input = torch.randn(128, 20)
            >>> output = m(input)
            >>> print(output.size())
            torch.Size([128, 30])
        """
        __constants__ = ['in_features', 'out_features']
    
        def __init__(self, in_features, out_features, bias=True):
            super(Linear, self).__init__()
            self.in_features = in_features
            self.out_features = out_features
            self.weight = Parameter(torch.Tensor(out_features, in_features))
            if bias:
                self.bias = Parameter(torch.Tensor(out_features))
            else:
                self.register_parameter('bias', None)
            self.reset_parameters()
    
        def reset_parameters(self):
            init.kaiming_uniform_(self.weight, a=math.sqrt(5))
            if self.bias is not None:
                fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
                bound = 1 / math.sqrt(fan_in)
                init.uniform_(self.bias, -bound, bound)
    
        def forward(self, input):
            return F.linear(input, self.weight, self.bias)
    
        def extra_repr(self):
            return 'in_features={}, out_features={}, bias={}'.format(
                self.in_features, self.out_features, self.bias is not None
            )

显然，在\_\_init\_\_(self, in\_features, out\_features, bias=True)中，下面的代码使用了nn.Parameter()对weights进行了初始化：

self.weight = Parameter(torch.Tensor(out_features, in_features))

--------------------

参考资料：  
[torch.nn.Parameter理解][torch.nn.Parameter]

[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5b-N6ICF44Gu5Lmx5aSq6YOO_size_20_color_FFFFFF_t_70_g_se_x_16_pic_center]: https://img-blog.csdnimg.cn/9ace0e8fa08749198f20825a610ff5ce.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBA5b-N6ICF44Gu5Lmx5aSq6YOO,size_20,color_FFFFFF,t_70,g_se,x_16#pic_center
[torch.nn.Parameter]: https://blog.csdn.net/qq_28753373/article/details/104179354