[MLE] Artificial Neural Network Training

Overview

  • Error Functions
  • Basic Linear Algebra
  • Singular Value Decomposition
  • Gradient Descent
  • Backpropagation
  • Deep Learning

Error Functions

In order to optimise the performance of ANNs an error function on the training set must be minimised
This is done by adjusting:

  • Weights connecting nodes
  • Network Architecture
  • Parameters of non-linear functions h(a)

Backpropagation

  • Used to calculate derivatives of error function efficiently
  • Error propagate backwards layer by layer

Iterative minimisation of error function:

  1. Calculate derivative of error function with respect to weights
  2. Derivatives used to adjust weights
    enter image description here

That’s the way we do backpropagation, but after get the derivatives, how do we update our weights?
Here is a graph I found on the internet:
enter image description here

In the lecture, it introduces as follows
enter image description here

Basic Linear Algebra

Matrix Determinant

  • Used in many calculations, e.g.
    • matrix inversion
    • singularity testing(singular iff |A| = 0)
    • det(A) = |A|
    • enter image description here

Eigenvalues

Given an invertible matrix M, an eigenvalue equation can be found in terms of a set of orthogonal vectors \(v_i\) and scalars \(\lambda_i\) such that \(Mv_i = \lambda_iv_i\)
Eigenvalues are found by solving the characteristic equation: \(| A - \lambda I| = 0\)
enter image description here

enter image description here

Jacobian and Hessian

enter image description here
enter image description here
enter image description here
Ans:13 and 33**
enter image description here

When doing BP, how to calculate gradient of error function

enter image description here

Regularization

We always add regularization in our neural network calculation
In CS231, we first know regularization from this slide:
enter image description here

Why use regularization

  • penalise bad weights
  • avoid overfitting
  • early stopping
    We use regularization to penalise large weights and unbalanced weights.
    Regularization is a technique used in an attempt to solve the overfitting problem in statistical models.
    And you might ask: OK I have everything now. How can I tune in the regularization term \(\lambda\)?
    One possible answer is to use cross-validation: you divide your training data, you train your model for a fixed value of \(\lambda\) and test it on the remaining subsets and repeat this procedure while varying \(\lambda\). Then you select the best \(\lambda\) that minimizes your loss function.
    Also, it is a way of early stopping since the test error will be steady.

评论

发表评论

此博客中的热门博文

[MLE] W2 Multivariate linear regression

图片
Multivariate linear regression What we have talked about in week 1 is linear regression with single feature. For example, in the house pricing predicting problem, we only use the size of house (x) to predict the price (y) . But what if there are other features like age, bedroom numbers and so on? Linear regression with multiple variables is also known as “multivariate linear regression” According to the graph above, we now have new representation symbols: n = number of features m = number of training examples \(x^{(i)}\) = \(i^{th}\) training example \(x^{(i)}_j\) = feature j in \(i^{th}\) training example 1416 is \(x_{1}^{(2)}\) in this case, m is 47 for example, n is 4. Hypothesis function The multivariable form of the hypothesis function accommodating these multiple features is as follows: if we make \(x_0\) to 1, the \(h_{\theta}(x)\) will be \(\theta^Tx\), which is \(\theta\) transpose matrix multiply by x, where \(\theta\) is a vector of 1 by(n+1) dimension an
阅读全文

[MLE] W1 Introduction

图片
Before get started The blog is based on the machine learning course on Coursera, taught by Andrew Ng . From today, i will begin my study trip on Machine learning, which will be labelled as MLE. This course will last 11 weeks and today is week 1. The application of machine learning is everywhere: Database mining Applications can’t program by hand Self-customizing program Understanding human learning(brain, real AI) Introduction What is Machine Learning Two definitions of machine learning are offered: Arthur Samuel described it as: “the field of study that gives computers the ability to learn without being explicitly programmed.” This is an older, informal definition. Tom Mitchell provides a more modern definition: “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.” If we raise a example of what is machine learning: If
阅读全文

[AIM] MetaHeuristics

图片
Day2 Outline Metaheuristics -definition Single point based search method(simple stochastic local search) Iterated local search Scheduling Tabu Search Metaheuristic what is a metaheuristic? A metaheuristic is a high-level problem independent algorithmic framework that provides a set of guidelines or strategies to develop heuristic optimization algorithms. what is the difference between heuristic and metaheuristic? Heuristic : Heuristics are problem-dependent techniques. As such, they usually are adapted to the problem at hand and they try to take full advantage of the particularities of this problem. However, because they are often too greedy, they usually get trapped in a local optimum and thus fail, in general, to obtain the global optimum solution. Metaheuristic : Meta-heuristics, on the other hand, are problem-independent techniques . As such, they do not take advantage of any specificity of the problem and, therefore, can be used as black boxes . In general, t
阅读全文