MikeChen's Blog

Overview Decision Trees Linear vs non-linear classifiers Entropy C4.5 Random Forests Some fundamental concepts The process of selecting a specific model, given a new input x, can be described by a sequential decision making process corresponding to the traversal of a binary tree (one that splits into two branches at each node). Here we focus on a particular tree-based framework called classification and regression trees Basic Decision Trees Decision trees apply a sequence of linear decisions, that often depend on only a single variable at a time. Such trees partition the input space into cuboid regions, gradually refining the level of detail of a decision until a leaf node has been reached, which provides the final predicted label As the above figure indicates, we should follow the rules for tree classification: Start at root node Evaluate relevant attribute of that node Follow branch according to attribute evaluation to descendent node Descendent node can then be co...

Linear Classification Discriminant functions Least squares for classification Fisher’s linear discriminant K-Nearest Neighbour(KNN) Classification What’s decision boundaries/linearly separable? The goal in classification is to take an input vector x and to assign it to one of K discrete classes \(C_k\) where k = 1,…,K. In the most common scenario, the classes are taken to be disjoint, so that each input is assigned to one and only one class. The input space is thereby divided into decision regions whose boundaries are called decision boundaries or decision surfaces. In this blog, we consider linear models for classification , by which we mean that the decision surfaces are linear functions of the input vector x and hence are defined by (D-1) -dimensional hyperplanes within the D-dimensional input space. For example: in 3D input space, the decision boundary will be a 2D surface What is Linear Separability Linearly separable data: Datasets whose classes can be separated by...

Table of Content Linear Regression with one variable(Univariate) Multivariate Linear Regression Gradient Descent Linear Regression with one variable So what is univariate linear regression? Let’s first look at one slide in the class: In my own words, let’s first imagine what is linear regression problem? House price prediction! Yes, it is a typical regression problem, and if we predict the house using linear algebra, like y = ax + b, it can be a linear regression problem. To apply our learning algorithm, we need to predict a function of house price first, and adjust it to resemble the actual function(training set). Thus, the h(x) is what we called hypothesis function , it is a hypothesis. And what we do is to modify the intrinsic parameters w or you can just say modify weights. Now we need to find a way to modify w so that our hypothesis function is more similar to the actual function, which comes to a new term cost function Cost function Cost function, in brief, is the...

Some content had wrote before but I would write it again for the completeness What’s Machine Learning? Two definitions of machine learning are offered: Arthur Samuel described it as: “the field of study that gives computers the ability to learn without being explicitly programmed.” This is an older, informal definition. Tom Mitchell provides a more modern definition: “ A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E .” Machine learning aim Learning functions: Given a set of training data consisting of pairs (x, f(x)) , where x are called features and f (x) = y the labels , Machine learning (ML) is the process of learning a function, or hypothesis, h(x) -> yˆ that best approximates the unknown function f(x) . This ability to make correct predictions on unseen data is called generalisation . Representation view on ML Learning...

Sorry for the delay and disappear in the summer. The reason is that Blogger has been blocked in China and even the VPN cannot load all the images and materials. Now from then on, I would again write something on module “Machine learning” again. Hopefully you will like it. To sum up what I got in summer: Machine learning related knowledge I’ve got: ・ Self learned whole Machine Learning course on coursera ・ Know basic concept of CNN, RNN, LSTM, GAN, etc. ・ Know the details of CNN and how to use them ・ Be able to use caffe framework(Did a small project about face detection using caffe in summer, which is a simple classification problem) ・ Be able to use pycaffe as the programming language.(but not matcaffe because I seldom use matlab) Also, I have implement an application of “image style-transfer” in summer according to the paper from CVPR 2015 What might be done in the future as the final year project, my research area is image super-resolution. I mig...

The problem of overfitting Consider the problem of predicting y from x ∈ R. The leftmost figure below shows the result of fitting a \(y = θ_0+θ_1x\) to a dataset . We see that the data doesn’t really lie on straight line, and so the fit is not very good. Instead, if we had added an extra feature \(x_2\) , and fit \(y=θ_0+θ_1x+θ_2x^2\) , then we obtain a slightly better fit to the data (See middle figure). Naively, it might seem that the more features we add, the better. However, there is also a danger in adding too many features: The rightmost figure is the result of fitting a 5th order polynomial y=\(θ_0+θ_1x+θ_2x^2+θ_3x^3+θ_4x^4\). We see that even though the fitted curve passes through the data perfectly, we would not expect this to be a very good predictor of, say, housing prices (y) for different living areas (x). Without formally defining what these terms mean, we’ll say the figure on the left shows an instance of underfitting —in which the data clearly shows structure not capt...

Neural Network Table of Contents: Quick intro without brain analogies Modeling one neuron Biological motivation and connections Single neuron as a linear classifier Commonly used activation functions Neural Network architectures Layer-wise organization Summary Additional references Quick intro It is possible to introduce neural networks without appealing to brain analogies. In the section on linear classification we computed scores for different visual categories given the image using the formula \( s = W x \), where \(W\) was a matrix and \(x\) was an input column vector containing all pixel data of the image. In the case of CIFAR-10, \(x\) is a [3072x1] column vector, and \(W\) is a [10x3072] matrix, so that the output scores is a vector of 10 class scores. An example neural network would instead compute \( s = W_2 \max(0, W_1 x) \). Here, \(W_1\) could be, for example, a [100x3072] matrix transforming the image into a 100-dimensional intermediate vector. The funct...