NOTES ON STATISTICS, PROBABILITY and MATHEMATICS

Perceptron:

In machine learning, the perceptron is an algorithm for supervised learning of binary classifiers: functions that can decide whether an input (represented by a vector of numbers) belongs to one class or another (Wikipedia). The idea is similar to logistic regression, although the optimization is different:

Vectorially, the \(d\) features or attributes of an example are \(\bf x\), and the idea is to “pass” the example if:

\(\displaystyle \sum_{1}^d \theta_i x_i > \text{theshold}\) or…

\(h(x) = \text{sign}\big(\displaystyle \sum_{1}^d \theta_i x_i - \text{theshold}\big)\). The sign function results in \(1\) or \(-1\), as opposed to \(0\) and \(1\) in logistic regression.

The threshold will be absorbed into the bias coefficient, \(+ \theta_0\). The formula is now:

\(h(x) = \text{sign}\big(\displaystyle \sum_0^d \theta_i x_i\big)\), or vectorized:

\(h(x) = \text{sign}(\theta^T\bf x)\).

Misclassified points will have:

\(\text{sign}(\theta^T\bf x) \neq y_n\), meaning that the dot product of \(theta\) and \(\bf x_n\) will be positive (vectors in the same direction), when \(y_n\) is negative, or the dot product will be negative (vectors in opposite directions), while \(y_n\) is positive:

The process starts with random weights or coefficients, and calculates for every missclassified points or examples \(n\) in the training sample:

\(\color{red}{\theta} := \color{blue}{\theta} + y_n \times \bf x_n\)

In this example I use logistic regression to get a decission boundary. We are looking at two test results, and the ultimate outcome of whether the student gets into college or not.

The code is as follows:

dat = read.csv("perceptron.txt", header=F)
colnames(dat) = c("test1","test2","y")
dat[1:5,]
##      test1    test2 y
## 1 34.62366 78.02469 0
## 2 30.28671 43.89500 0
## 3 35.84741 72.90220 0
## 4 60.18260 86.30855 1
## 5 79.03274 75.34438 1
plot(test2 ~ test1, col = as.factor(y), pch = 20, data=dat,
     main = "Decision Boundary - College Admission")

fit = glm(y ~ test1 + test2, family = "binomial", data = dat)
coefs = coef(fit)
x = c(min(dat[,1])-2,  max(dat[,1])+2)
y = c((-1/coefs[3]) * (coefs[2] * x + coefs[1]))
lines(x, y, lwd = 3, col = rgb(0,.9,.1,.4))

The boundary decision line corresponds to:

\(0 = \theta_0 + \theta_1 \times \text{test1} + \theta_2 \times \text{test2}\). Hence, \(\text{test2}=(\frac{-1}{\theta_2})\times (\theta_0 + \theta_1 \times \text{test1} ).\)

Home Page

NOTE: These are tentative notes on different topics for personal use - expect mistakes and misunderstandings.