L1和L2正则化理解

1.从拉格朗日乘数法理解
$$\begin{gathered} minJ(W,b,x) \\ s.t||W|{|}_1-C\le 0 \end{gathered}$$$$\begin{gathered} L(W,\lambda )=J(W)+\lambda (||W|{|}_1-C) \\ \underset{W}{\min }\underset{\lambda }{\max }L(W,\lambda ) \\ s.t.\lambda \ge 0 \end{gathered}$$
$$\begin{array}{rl}{L}_1(W,\lambda )& =J(W)+\lambda (||W|{|}_2-C)\\ & =J(W)+\lambda ||W|{|}_2-\lambda C\end{array}$$
$$\begin{array}{rl}{L}_2(W,\lambda )& =L_1(W,\lambda )+\lambda C\\ & =J(W)+\lambda ||W|{|}_2\end{array}$$
$$\begin{gathered} {\arg }_W(\underset{W}{\min }\underset{\lambda }{\max }{L}_1(W，\lambda ) \\ s.t.\lambda \ge 0) \\ =ar{g}_W(\underset{W}{\min }\underset{\lambda }{\max }{L}_2(W,\lambda ) \\ s.t.\lambda \ge 0) \\ 这两个问题等价，因为L1和L2对W求导等于0的极值点相同L_2没有C半径可以任意大小。 \\ \lambda 用于调节梯度 \\ 通过图可以可看出L1稀疏 \end{gathered}$$
2.权重衰减
$$\begin{gathered} 损失函数：J(W,b) \\ 权重更新:W=W-\eta \cdot {▽}_{W}J(W) \end{gathered}$$
$$\begin{gathered} 正则化后：\begin{array}{rl}& \\ 损失函数\hat{J}(W)& \\ & =J(W)+\lambda ||W|{|}_2\\ & =J(W)+\frac{\alpha }{2}{W}^{T}W\end{array}权重更新： \\ \begin{array}{rl}W& =W-\eta \cdot {▽}_W\hat{J}(W)\\ & =W-\eta \cdot {▽}_{w}J(W)-\eta \cdot \alpha W(最后一项没有带三角w，已经求过导了）\\ & =(1-\eta \cdot \alpha )W-\eta \cdot {▽}_{W}J(W)\end{array} \\ 从最后一行公式可以看出，学习率的范围是0到1，进行了权重的限制 \end{gathered}$$
3.从贝叶斯概率理解
$$\begin{gathered} 似然是似然函数的简称 \\ 最大似然值是\theta 最大的取值 \\ 贝叶斯公式：P(\theta |X)=\frac{P(X|\theta )}{P(X)}\cdot P(\theta ) \\ P(\theta )是先验概率 \\ P(\theta |X)是后验概率 \\ {L}_x(\theta )=L(\theta |X)=P(X|\theta ) \\ P(\theta |X)=\frac{L(\theta |X)}{P(X)}\cdot P(\theta ) \\ {f}_x(\theta )=P(\theta |X)=\frac{L(\theta |X)}{P(X)}\cdot P(\theta ) \\ {f}_x(\theta )\propto P(X|\theta )\cdot P(\theta )=L(\theta |X)\cdot P(\theta ) \\ 最大似然估计（MLE） \\ 认定：\hat{\theta }=argma{x}_{\theta }L(\theta |X) \\ 最大后验估计(MAP) \\ 认定：\theta =argma{x}_{\theta }(L(\theta |X)\cdot P(\theta )) \end{gathered}$$
有时间继续补