文章目录

• Terminology
• Neural Networks: Representation
• • Motivations
  • • Non-linear Hypotheses
    • Neurons and the Brain
  • Neural Networks
  • • Model Representation
    • • Neuron in the brain
      • Neuron model: Logistic unit
      • Neural Network
      • • computations
      • Forward propagation: Vectorized implementation
      • • Element-wise
        • Vectorized implementation
      • Neural Network learning its own features
      • Other network architectures
  • Applications
  • • Examples and Intuitions I
    • • Non-linear classification example: XOR/XNOR
      • Simple example: AND
      • Example: OR function
    • Examples and Intuitions II
    • • Negation(NOT)
      • Putting it together: x1 XNOR x2
    • Multiclass Classification
    • • Multiple output units: One-vs-all

Terminology

asymptotically: 渐近地

resurgence: 再起,复活

auditory cortex: 听觉皮层

somatosensory cortex: 体感皮层

jam-packed: 塞满的

dendrites and axon: 树突和轴突

bias unit: 偏置单元

forward propagation: 前向传播

XOR: 异或

XNOR: 同或

negation: 反面,对立面;否定;拒绝

Neural Networks: Representation

Motivations

Non-linear Hypotheses

the dimension of feature size is too large

有10000个像素点，最后结果取$C_{10000}^2$，$10000^2/2=5\times 10^7$

Performing linear regression with a complex set of data with many features is very unwieldy. Say you wanted to create a hypothesis from three (3) features that included all the quadratic terms:
$$g({\theta }_0+{\theta }_1{x}_1^2+{\theta }_2{x}_1{x}_2+{\theta }_3{x}_1{x}_3+{\theta }_4{x}_2^2+{\theta }_5{x}_2{x}_3+{\theta }_6{x}_3^2)$$
That gives us 6 features. The exact way to calculate how many features for all polynomial terms is the combination function with repetition: http://www.mathsisfun.com/combinatorics/combinations-permutations.html $\frac{(n+r-1)!}{r!(n-1)!}$. In this case we are taking all two-element combinations of three features: $\frac{(3+2-1)!}{(2!\cdot (3-1)!)}=6.$

For 100 features, if we wanted to make them quadratic we would get $\frac{(100+2-1)!}{(2\cdot (100-1)!)}=5050$ resulting new features.

We can approximate the growth of the number of new features we get with all quadratic terms with $\mathcal{O}(n^2/2)$. And if you wanted to include all cubic terms in your hypothesis, the features would grow asymptotically at $\mathcal{O}(n^3)$. These are very steep growths, so as the number of our features increase, the number of quadratic or cubic features increase very rapidly and becomes quickly impractical.

Example: let our training set be a collection of 50 x 50 pixel black-and-white photographs, and our goal will be to classify which ones are photos of cars. Our feature set size is then n = 2500 if we compare every pair of pixels.

Now let’s say we need to make a quadratic hypothesis function. With quadratic features, our growth is $\mathcal{O}(n^2/2)$. So our total features will be about $2500^2/2=3125000$, which is very impractical.

Neural networks offers an alternate way to perform machine learning when we have complex hypotheses with many features.

Neurons and the Brain

Neural Networks

Model Representation

Neuron in the brain

Neuron model: Logistic unit

bias unit: $x_0$ 偏置项，视情况而定

activation function: 激励函数

weights: 权重，等同于之前所谈的parameters，即 $\theta$

At a very simple level, neurons are basically computational units that take inputs (dendrites) as electrical inputs (called “spikes”) that are channeled to outputs (axons). In our model, our dendrites are like the input features $x_1\cdots x_n$, and the output is the result of our hypothesis function. In this model our $x_0$ input node is sometimes called the “bias unit.” It is always equal to 1. In neural networks, we use the same logistic function as in classification, $\frac{1}{1+e^{-{\theta }^{T}x}}$, yet we sometimes call it a sigmoid (logistic) activation function. In this situation, our “theta” parameters are sometimes called “weights”.

Neural Network

input layer

hidden layer

output layer

computations

The values for each of the “activation” nodes is obtained as follows:
$$\begin{gathered} a_1^{(2)}=g({\Theta }_{10}^{(1)}{x}_0+{\Theta }_{11}^{(1)}{x}_1+{\Theta }_{12}^{(1)}{x}_2+{\Theta }_{13}^{(1)}{x}_3) \\ {a}_2^{(2)}=g({\Theta }_{20}^{(1)}{x}_0+{\Theta }_{21}^{(1)}{x}_1+{\Theta }_{22}^{(1)}{x}_2+{\Theta }_{23}^{(1)}{x}_3) \\ {a}_3^{(2)}=g({\Theta }_{30}^{(1)}{x}_0+{\Theta }_{31}^{(1)}{x}_1+{\Theta }_{32}^{(1)}{x}_2+{\Theta }_{33}^{(1)}{x}_3) \\ {h}_{\Theta }(x)=a_1^{(3)}=g({\Theta }_{10}^{(2)}{a}_0^2+{\Theta }_{11}^{(2)}{a}_1^2+{\Theta }_{12}^{(2)}{a}_2^2+{\Theta }_{13}^{(2)}{a}_3^2) \end{gathered}$$
${\Theta }^{(1)}$ 是layer1的权重矩阵，它是一个3x4的矩阵（3行4列）。每一行权值乘上输入的值，就能得到一个激活借点的值。

此处，$h_{\Theta }(x)$是计算激活节点值之和的Logistic Function，该函数已乘以另一个参数矩阵 ${\Theta }^{(2)}$ ，包含第二层节点的权重。

每层都有该层对应的权重矩阵，这些权重矩阵的维度定义如下：

If network has $s_j$ units in layer $j$ and $s_{j+1}$ units in layer $j+1$, then ${\Theta }^{(j)}$ will be of dimension $s_{j+1}\times (s_j+1)$.

例如下图：layer1有2个输入节点，layer2有4个激活节点，那么${\Theta }^{(1)}$的尺寸为4x3。因为$x_0$作为偏置项也得加上，所以要进行4行3列的映射。

Forward propagation: Vectorized implementation

Element-wise

element-wise 是两个张量之间的操作，它在相应张量内的对应的元素进行操作。

例如$v$表示对每一个输入向量 $v⨀w=s$ 乘以一个给定的”权重"$w$向量。换句话说，就是通过一个乘子对数据集的每一列进行缩放。这个转换可以表示为如下的形式：
{ v 1 , v 2 , v 3 } T ⨀ { w 1 , w 2 , w 3 } T = { v 1 w 1 , v 2 w 2 , v 3 w 3 } T \{v_1,v_2,v_3 \}^{T}\bigodot \{w_1,w_2,w_3 \}^{T}=\{v_1w_1,v_2w_2,v_3w_3 \}^{T} {v1​,v2​,v3​}T⨀{w1​,w2​,w3​}T={v1​w1​,v2​w2​,v3​w3​}T

简而言之，同位元素对应相乘/向量的分量对应分别相乘。

Vectorized implementation

把 $g({\Theta }_{10}^{(1)}{x}_0+{\Theta }_{11}^{(1)}{x}_1+{\Theta }_{12}^{(1)}{x}_2+{\Theta }_{13}^{(1)}{x}_3)$ 括号里的写成$z_1^{(2)}$，上标2代表layer2。

$z^{(2)}$ 可以表示为权重矩阵${\Theta }_{(1)}$乘$x$，这里的x为了统一形式可以记作layer1激活节点$a^{(1)}$。计算对应的$a^{(2)}$即为$g(z^{(2)})$，这里计算是elememt-wise的。

layer2增加一个偏置项$a_0^{(2)}=1$，$z^{(3)}$可以表示为权重矩阵${\Theta }^{(2)}$乘矩阵$a^{(2)}$。故最后的$h_{\Theta }(x)=a^{(3)}=g(z^{(3)})$

In this section we’ll do a vectorized implementation of the above functions. We’re going to define a new variable $z_k^{(j)}$ that encompasses the parameters inside our g function. In our previous example if we replaced by the variable z for all the parameters we would get:
$$\begin{gathered} a_1^{(2)}=g(z_1^{(2)}) \\ {a}_2^{(2)}=g(z_2^{(2)}) \\ {a}_3^{(2)}=g(z_3^{(2)}) \end{gathered}$$
In other words, for layer j=2 and node k, the variable z will be:
$$z_k^{(2)}={\Theta }_{k,0}^{(1)}{x}_0+{\Theta }_{k,1}^{(1)}{x}_1+\cdots +{\Theta }_{k,n}^{(1)}{x}_n$$
The vector representation of x and $z^j$ is:
$$x=\left[\begin{array}{c}{x}_0\\ x_1\\ \cdots \\ x_n\end{array}\right]$$
$$z^{(j)}=\left[\begin{array}{c}{z}_1^{(j)}\\ z_2^{(j)}\\ \cdots \\ z_n^{(j)}\end{array}\right]$$
Setting $x=a^{(1)}$, we can rewrite the equation as:
$$z^{(j)}={\Theta }^{(j-1)}{a}^{(j-1)}$$
We are multiplying our matrix ${\Theta }^{(j-1)}$ with dimensions $s_j\times (n+1)$ (where $s_j$ is the number of our activation nodes) by our vector $a^{(j-1)}$ with height (n+1). This gives us our vector $z^{(j)}$ with height $s_j$. Now we can get a vector of our activation nodes for layer j as follows:
$$a^{(j)}=g(z^{(j)})$$
Where our function g can be applied element-wise to our vector $z^{(j)}$.

We can then add a bias unit (equal to 1) to layer j after we have computed $a^{(j)}$. This will be element $a_0^{(j)}$ and will be equal to 1. To compute our final hypothesis, let’s first compute another z vector:
$$z^{(j+1)}={\Theta }^{(j)}{a}^{(j)}$$
We get this final z vector by multiplying the next theta matrix after ${\Theta }^{(j-1)}$ with the values of all the activation nodes we just got. This last theta matrix ${\Theta }^{(j)}$ will have only one row which is multiplied by one column $a^{(j)}$ so that our result is a single number. We then get our final result with:
$$h_{\Theta }(x)=a^{(j+1)}=g(z^{(j+1)})$$
Notice that in this last step, between layer j and layer j+1, we are doing exactly the same thing as we did in logistic regression. Adding all these intermediate layers in neural networks allows us to more elegantly produce interesting and more complex non-linear hypotheses.

Neural Network learning its own features

注意公式的蓝色部分，这看起来就像logistic regression，只是 $\theta$ 变成了 $\Theta$ 。而神经网络的独到之处，在于这里的$a^{(2)}$不是固定输入的，而是由固定输入的$x_1,x_2,x_3$学习而来的。

Other network architectures

You can have neural networks with other types of diagrams as well, and the way that neural networks are connected, that’s called the architecture.

Applications

Examples and Intuitions I

Non-linear classification example: XOR/XNOR

XOR：异或，x1和x2不同，y=1；否则y=0

XNOR：同或， x1和x2相同，y=1（上图中红叉）；否则y=0（上图中红圈）

XNOR 等价于 NOT XOR

Simple example: AND

Example: OR function

Examples and Intuitions II

Negation(NOT)

Putting it together: x1 XNOR x2

hidden layer的复杂组合，能够使得神经网络计算复杂的非线性函数

Multiclass Classification

Multiple output units: One-vs-all

So this is just like the “one versus all” method that we talked about when we were describing logistic regression, and here we have essentially four logistic regression classifiers, each of which is trying to recognize one of the four classes that we want to distinguish amongst.

以前的lable y都是写成数值1,2,3,4这样的形式，现在换成矩阵[1,0,0,0]这样的。$x^{(i)}$表示输入进去的图片，$y^{(i)}$用矩阵表示相应的行人/汽车/摩托车/卡车

我们希望找到一种方法，让神经网络输出这样的值。