Deep learning

The function outputs a scalar

Model → $y = b + wx_1$ (linear model)

Model → $y = b + \sum{wx_1}$ (increase domain knowledge)

The function outputs a correct given options(classes)

• Softmax

  $y^\prime = softmax(y)$ (Normalize y to 0 < $y^\prime$ < 1)

  $y^\prime = \dfrac{exp(y_i)}{\sum\limits_j{exp(y_i)}}$

  $\sum\limits_jy_i^\prime = 1$

  

• Mean Square Error

  $e = \sum\limits_i{(\^y_i - y_i^\prime)^2}$

• Cross-entropy

  $e = - \sum\limits_i{\^y_i ln(y_i^\prime)}$

  Best suit for classification

Minimize cross-entropy == maximize likelihood

$softmax$ == 2 input $sigmoid$

A function of parameters. How good a set of value is.

$\^y$ → label

$e_i = |y_i -\^y_i|$ → Mean Absolute Error

$e_i = (y_i -\^y_i)^2$ → Mean Square Error

$L(b, w) = \dfrac{1}{N}\sum{e_i}$ → loss(small the best)

• Gradient Descent

  $\theta^* = \min{L(\theta)}$

  $\eta$ → learning rate. (hyperparameter)

  $\eta \dfrac{\delta L}{\delta \theta}$ → Gradient Descent(only find local minima)

  $\theta^1 = \theta^0 - \eta \dfrac{\delta L}{\delta \theta}|_{\theta=\theta^0}$ (reverse direction of gradient)

  $g = \nabla L(\theta^0)$

  If $gradient$ is negative, increase $\theta$.

  $\theta^1 = \theta^0 - \eta g$

  Hyperparameter: defined by ourselves.

  epoch = N $\times$ Batch Updates

Critical points

Find loss function near $\theta = \theta^ \prime$

Tayler series approximation

$L(\theta) \approx L(\theta^ \prime) + (\theta - \theta^ \prime)^T g + \dfrac{1}{2} (\theta - \theta^ \prime)^T H(\theta - \theta^ \prime)$

$L(\theta) \approx L(\theta^ \prime) + v^T g + \dfrac{1}{2} v^T Hv$

• $g$ → gradient vector

  $g = \nabla L(\theta^ \prime)$ $g_i = \dfrac{\delta L(\theta^ \prime)}{\delta \theta_i}$

  If == 0 → at critical point

• $H$ → Hessian matrix

  $H_{ij} = \dfrac{\delta^2}{\delta \theta_i \delta \theta_j} L(\theta^ \prime)$

  Tells the properties of critical point & update direction

• Local minima

  $v^T Hv < 0$, for all $v$

  H is positive defined → All eigen values are positive

  

• Local maxima

  $v^T Hv > 0$, for all $v$

  H is negative defined → All eigen values are negative

  

• Saddle point

  else

  

  $u^THu$ = $u^T(\lambda u)$ = $\lambda ||u||^2$

  $\theta - \theta^ \prime = u$ $\theta = \theta^ \prime + u$

  Update direction: $(\lambda_i < 0) → u_i$

💡

Eigen value $\lambda$ → $\det({A - \lambda I}) = 0$
Eigen vector
$u_i$ → $(A - \lambda_i I) u_i = 0$

Batch (a hyperparameter)

epoch = see all batches once → shuffle after each epoch(divide to batches)

• Batch size = N (full batch)
  • long time for 1 update
  • Faster for 1 epoch

• Batch size = 1
  • Noisy, more updates
  • Faster for 1 update
  • Noise is better for training & testing

For parallel computing

Momentum

Mimic real world physics

Movement($m$): movement of last step - gradient at present, $m^0 = 0$

$m^1 = \lambda m^0 - \eta g^0$ (movement) → (sum of all past gradients)

$\theta^1 = \theta^0 + m^1$ (move to)

Batch Normalization

Smooth error surface

$w_i + \Delta w_i → L + \Delta L$

large $x_i$ has greater affect

• Feature normalization

  • $i$ → dimension

  • $\mu_i$ → mean

  • $\sigma$ → standard deviation

    $\~x_i^r = \dfrac{x_i^r - \mu_i}{\sigma_i}$ (all dimension are 0 with variance 1)

  

  If desired output ≠ 0 → add network parameter

  $\^x_i = \gamma \odot \~x_i + \beta$

  • $\odot$ → element wise multiplication

  • $\gamma$ → initially a 1 vector (until a good error surface is found)

  • $\beta$ → initially a 0 vector (until a good error surface is found)

  • Testing stage

    Moving average of training

    $\~x_i^r = \dfrac{x_i^r - \bar \mu_i}{\bar \sigma_i}$

    $\bar \mu = p \bar \mu + (1 - p) \mu^t$

• Linear

• Piecewise Linear - sets of sigmoid(activation) functions {Neuron}

$y_n = c \cdot sigmoid(b+wx_i) = c \dfrac{1}{1 + e^{-b+wx_n}}$

$y_n = b + \sum{c_i \cdot sigmoid(b_i + w_ix_n)}$

• $y_n = b + \sum\limits_{i}{c_i \cdot sigmoid(b_i + \sum\limits_{j}{w_{ij}x_j})}$

  • $i$ → piecewise sigmoid function

  • $j$ → range of knowledge domain

1 layer

Feature: $x$

Unknown parameter ($\theta$): $y, b, c^T, w$

$y_n = b + \sum\limits_{i}{c_i \cdot sigmoid(b_i + \sum\limits_{j}{w_{ij}x_j})}$

$y_n = b + \sum\limits_{2i}{c_i \cdot max(0, b_i + \sum\limits_{j}{w_{ij}x_j})}$ → ReLU (better)

An efficient Gradient Descent

• Chain Rule

  $y = g(x) z = h(y)$

  $\dfrac{dz}{dx} = \dfrac{dz}{dy} \dfrac{dy}{dx}$

  $\Delta x \rightarrow \Delta y \rightarrow \Delta z$

  $x = g(s) y = h(s) z = k(x, y)$

  $\dfrac{dz}{ds} = \dfrac{\delta z}{\delta x} \dfrac{dx}{ds} + \dfrac{\delta z}{\delta y} \dfrac{dy}{ds}$

  $\Delta x$

  $\Delta s$ $\Delta z$

  $\Delta y$

$C^n$ → distance between $y^n$ & $\^y^n$

$L(\theta) = \sum\limits_{n=1}^N{C^n(\theta)}$

$\dfrac{\delta L(\theta)}{\delta w} = \sum\limits_{n=1}^N {\dfrac{\delta C^n(\theta)}{\delta w}}$

$\dfrac{\delta C}{\delta w} = \dfrac{\delta z}{\delta w} \dfrac{\delta C}{\delta z}$

• Forward pass

  $\dfrac{\delta z}{\delta w} = x$ (input)

  $z = x_1w_1 + x_2w_2 + b$

• Backward pass

  $\dfrac{\delta C}{\delta z} = \dfrac{\delta a}{\delta z} \dfrac{\delta C}{\delta a}$

  $a = \sigma(z)$ activation function

  

  $\dfrac{\delta C}{\delta a} = \dfrac{\delta z^’}{\delta a} \dfrac{\delta C}{\delta z^’} + \dfrac{\delta z^{’’}}{\delta a} \dfrac{\delta C}{\delta z^{’’}}$

  $\dfrac{\delta z^’}{\delta a} = w$

  $\dfrac{\delta C}{\delta z} = \sigma^’(z) [w_3 \dfrac{\delta C}{\delta z^’} + w_4 \dfrac{\delta C}{\delta z^{’’}}]$

  $\sigma^’(z$) is a constant

  $\dfrac{\delta C}{\delta z^’} = \dfrac{\delta y_1}{\delta z^’} \dfrac{\delta C}{\delta y_1}$ $\dfrac{\delta C}{\delta z^{’’}} = \dfrac{\delta y_2}{\delta z^{’’}} \dfrac{\delta C}{\delta y_2}$

Observe training data first, then testing data.

Model bias

$y = b + wx_1$

• More features: increase domain knowledge $y = b + \sum{wx_1}$

• More layers: deep learning $y_n = b + \sum\limits_{i}{c_i \cdot sigmoid(b_i + \sum\limits_{j}{w_{ij}x_j})}$

Bad optimization

Big training data loss

• Gain insight from shallow network optimization

• Increase training data

• Data augmentation(隆乳): generate new data from existing data

• Constrained model: based on our interpretation of the problem

Mismatch

Distribution of training & testing data is different