[MLE] Linear Regression

Table of Content

Linear Regression with one variable

So what is univariate linear regression? Let’s first look at one slide in the class:
enter image description here
In my own words, let’s first imagine what is linear regression problem?
House price prediction!
Yes, it is a typical regression problem, and if we predict the house using linear algebra, like y = ax + b, it can be a linear regression problem.
To apply our learning algorithm, we need to predict a function of house price first, and adjust it to resemble the actual function(training set). Thus, the h(x) is what we called hypothesis function, it is a hypothesis. And what we do is to modify the intrinsic parameters w or you can just say modify weights.
Now we need to find a way to modify w so that our hypothesis function is more similar to the actual function, which comes to a new term cost function

Cost function

enter image description here
Cost function, in brief, is the sum of the difference between all training examples(points) and its predicted value.
enter image description here
For example, on the above image, when size is 750, the actual price is 200 but the predicted value is 150. Therefore, the cost or the difference is 50 in this point, and we sum all these cost.

Tricky point, why we have a \(\frac{1}{2n}\) before our \(\sum\) (sum)?

\(\frac{1}{n}\) is because we want to normalize them, which indicates that cost function is independent from training example size
\(\frac{1}{2}\) is because we will use Gradient Descent to calculate the minimal value of our cost, and when using partial derivatives, the square will times this \(\frac{1}{2}\) for easier calculation

Other cost functions

Well, this cost function is called L2-norm. There are also L1-norm but I think it is worse.
| enter image description here

Multivariate Linear Regression

In its simplest form, multivariate regression is simply the linear sum of the multiplications of each of the features with their corresponding weight term:
Multivariate linear regression with first-order polynomial:
\(yˆ=h(x,w)=w_0 +w_1x_1 +...+w_jx_j +...+w_dx_d\)
Higher-order polynomial:
enter image description here
Actually, I think the explanation by Andrew Ng in Coursera is better, See here

Gradient Descent

Cool, here comes to my favorite gradient descent, it is all about math to calculate the minima of cost function
We want to minimise J(w0, w1, … wk)
  • First start with some initial values for w0, w1, …wk
  • Keep updating W to reduce J(w0, w1, … wk), hoping to end in the minimum
enter image description here
Take care! This simultaneous updates, or that would be wrong because you use the updated w0 to calculate new temp1
enter image description here

评论

发表评论

此博客中的热门博文

[MLE] W2 Multivariate linear regression

图片
Multivariate linear regression What we have talked about in week 1 is linear regression with single feature. For example, in the house pricing predicting problem, we only use the size of house (x) to predict the price (y) . But what if there are other features like age, bedroom numbers and so on? Linear regression with multiple variables is also known as “multivariate linear regression” According to the graph above, we now have new representation symbols: n = number of features m = number of training examples \(x^{(i)}\) = \(i^{th}\) training example \(x^{(i)}_j\) = feature j in \(i^{th}\) training example 1416 is \(x_{1}^{(2)}\) in this case, m is 47 for example, n is 4. Hypothesis function The multivariable form of the hypothesis function accommodating these multiple features is as follows: if we make \(x_0\) to 1, the \(h_{\theta}(x)\) will be \(\theta^Tx\), which is \(\theta\) transpose matrix multiply by x, where \(\theta\) is a vector of 1 by(n+1) dimension an...
阅读全文

[MLE] W1 Introduction

图片
Before get started The blog is based on the machine learning course on Coursera, taught by Andrew Ng . From today, i will begin my study trip on Machine learning, which will be labelled as MLE. This course will last 11 weeks and today is week 1. The application of machine learning is everywhere: Database mining Applications can’t program by hand Self-customizing program Understanding human learning(brain, real AI) Introduction What is Machine Learning Two definitions of machine learning are offered: Arthur Samuel described it as: “the field of study that gives computers the ability to learn without being explicitly programmed.” This is an older, informal definition. Tom Mitchell provides a more modern definition: “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.” If we raise a example of what is machine learning: If...
阅读全文

[AIM] MetaHeuristics

图片
Day2 Outline Metaheuristics -definition Single point based search method(simple stochastic local search) Iterated local search Scheduling Tabu Search Metaheuristic what is a metaheuristic? A metaheuristic is a high-level problem independent algorithmic framework that provides a set of guidelines or strategies to develop heuristic optimization algorithms. what is the difference between heuristic and metaheuristic? Heuristic : Heuristics are problem-dependent techniques. As such, they usually are adapted to the problem at hand and they try to take full advantage of the particularities of this problem. However, because they are often too greedy, they usually get trapped in a local optimum and thus fail, in general, to obtain the global optimum solution. Metaheuristic : Meta-heuristics, on the other hand, are problem-independent techniques . As such, they do not take advantage of any specificity of the problem and, therefore, can be used as black boxes . In general, t...
阅读全文