Linear Regression Simplified with Mathematics

Note:

Don't be stressed out with any python code, this is a mathematics tutorial, so just focus on mathematics and build your intuition.

Machine Learning

Artificial Intelligence needs no introduction for us programmers. But when most people talk about AI, they imagine an intelligent machine capable of learning or thinking like we humans do. While to some extent it's true, but it's wrong to say it's completely right. AI simply is a result of some mathematical and statistical algorithms and today we'll learn the basics of how these algorithm works.

Let's talk about what machine learning is. The set of mathematical algorithms which our AI systems use behind the curtains is called the machine learning.

So, machine learning is the process of how our computers gets trained to learn like humans while AI (in general) is the end result.

There are (in general) three types of Machine Learning Techniques:

• Supervised Learning
• Unsupervised Learning
• Re-inforcement Learning

Supervised Learning:

• Regression - Predict numerical outputs
• Classification - Predict categorical outputs (yes/no)

Re-inforcement Learning:

To simplify: The robots that you see over on youtube videos or maybe a game AI that's so hard to beat. For example, in chess there're different difficulty levels (advanced AI) which could scare best among the best of chess players. Its working is based on mathematical agents (algorithm of action, state, reward) which is really not the focus of today's tutorial.

Anyhow, we don't usually use reinforcement learning, so let's talk about what we use 90% of the time i.e. Supervised and Unsupervised Learning. And believe it or not, it all starts from just a mathematical line called regression.

Even if we talk about deep learning with deep neural networks, it all starts with a simple line equation. So, let's understand what line equation is and how it helps to make predictions using Regression.

Intuition based Regression model

Let's imagine a hypothetical case where our objective is to predict salaries of employees based on their work experience. To simplify things, we will assume that salary increases as work experience increase.

So, now assume that in this hypothetical case, we have past data of 3 employees.

So, now look at the data and now look at the visualization below. Again, don't worry about the code.

In this case, the data has 3 training examples, that is, we have data of 3 employees. Now, our problem statement is to predict salaries of new employees based on this data.

Based on our intuition, we can look at the plot and imagine that if an employee has 4 years of experience, their salary should be 4000, right.

This black line also represents our hypothesis that the data is linear and confirms for us that if an employee has 4 years of work experience their salary should be 4000

Locate the green point on the plot which is located on the x axis (work experience in years)

Now start from the green point on x axis (work experience data point 4) move upwards vertically while following the blue line till it hits the black line.

Now, you've reached the red point.

Now, follow the blue line horizontally to its left.

Now, you've reached the yellow point. This yellow point tells the salary (4000) on the y axis of the employee with 4 years of work experience.

Congrats, you've made your first intuition based machine learning system which predicts employee salaries.

Now, if an employee with 5 years of work experience comes, you will be able to tell that their salary should be 5000.

In this hypothetical case, the past data of employees that we used was linear in nature which means there exists a straight line which could represent the whole dataset.

In real world though, our data may or may not be linear in nature. But for simplicity sake, we'll assume data is linear and continue to learn our first machine learning algorithm.

What we've done so far in machine learning terms is called a hypothesis, an assumption that we make from the behavior of our data, that is in this case, our assumption was that data is linear.

Line Equation:

For those who already are well versed with line equation, you could choose to skip to Mathematical model of Regression.

Line equation is given by $$y=ax+b$$

Line: Take any two points, join them, extend boundaries and you have a line. Let's take two points as (0, 0) and (10, 10). If we join them, we'll have this line.

Another way to make a line is by using the equation y=ax+b.

Here, b is the y-intercept or the point where the line meets y axis.

a is the slope of the line.

Using just a and b, we can draw any line. The previous line has slope=1, yintercept=0. Let's try with a slope=2, yintercept=1.

Look how slanted the line has become.

Conversely, to calculate the slope (a) of the line y=ax+b, we can choose any two points on the line. Let's select (0,1) and (5, 11)

$$a = \frac{y2 - y1}{x2 - x1}$$

$$a = \frac{11 - 1}{5 - 0}$$

$$a = 2$$

$$b = 1 $$

y=2x+1

Mathematical model of Regression

Let's understand the mathematical model of machine learning now (Regression in this case). Linear relationship is mathematically expressed as:

• Line equation:

$$y=ax+b$$

Where, a and b are some parameters which can change the behavior of our line

a is the slope of the line

b is called the y intercept

We are already well versed with the line equation. Check out a line equation tutorial if you aren't yet comfortable with the line equation. But for those who remember this, let's proceed.

$$y=mx+c$$

Line equation can take up any form

$$y=mx+c$$ $$or$$ $$y=ax+b$$ $$or$$ In machine learning terms the equation is

$$h\theta (x) = \theta0 + \theta_1 x$$

In this lecture, we'll assume the terminology

$$y=ax+b$$

Each and every machine learning algorithm follows more or less the same process.

First we make a hypothesis, then we test it using some methods if its making correct predictions or not, and if not, then we minimize the errors

Procedure:

1 Hypothesis: y=ax+b

2 Hyper-parameters: a, b

With respect to different hyper-parameters, nature of the line / hypothesis changes

3 Cost function: Cost function is how we evaluate the error with respect to a, b

For first time readers, the equation or its notation might look complex, so stick to the process I'll explain everything in it's own sweet time.

MSE: Mean Squared Error

$$error(a,b) = \frac{1}{2m} \sum\limits_{i=0}^{m-1} (predictedValue^{(i)} - actualValue^{(i)} )^2$$

where, m: total training examples

i: 0,1,2: (index of an employee) In our problem statement, there are 3 training examples, so (i) could take values 0, 1 or 2. Employee0, Employee1, Employee2. Remember, indexing in python starts at 0, so we're using 0,1,2 as just an analogy.

4 Objective: minimize error(a,b) subjected to the hyper-parameters (a,b)

Mean Squared Error

$$error(a,b) = \frac{1}{2m} \sum\limits_{i=0}^{m-1} (predictedValue^{(i)} - actualValue^{(i)} )^2$$

where, m = 3: total training examples i = 0, 1, 2

To further simplify things, we'll use the terminology that, predictions made by the black line for the training examples are represented as pred(i) i.e.

pred0, pred1, pred2 for predicted salary of employee0, employee1, employee2 respectively

While actual salary values will be represented as actual(i) i.e.

actual0, actual1, actual2 for actual salary of employee0, employee1, employee2 respectively

In our example, we have past data of 3 employees, so we calcuate error as

$$error = \frac{1}{2*3} ((pred0 - actual0)^2 + (pred1 - actual1)^2 + (pred2 - actual2)^2)$$

The actual values for the employee salary were 1000, 2000, 3000 and respectively their salary predicted by the black line were also 1000, 2000, 3000

$$error = \frac{1}{2*3} ((1000 - 1000)^2 + (2000 - 2000)^2 + (3000 - 3000)^2 )$$

$$error = \frac{1}{6} (0 + 0 + 0)$$

$$error = 0$$

Deep dive into basics of mathematical Regression

In this case we already knew the correct hyper-parameters (a,b) (a=1000, b=0) for the line equation (y=1000x). But, usually our data is not so simple that we could correctly guess the hyper-parameters (a,b). There could be various other reasons that might constrain us from reaching correct hyper-parameters. We'll get into those details later on.

• What usually happens in machine learning is that we first assume random set of hyper-parameters.

• Then, we use cost-function to calculate the error subjected to the chosen hyper-parameters.

• Lastly, we use an optimization algorithm like gradient-descent to minimise the error. But, for simplicity sake, we'll manually tweak hyper-parameters to reduce error without using gradient descent.

First set of hyper-parameters a=0, b=0

We are already comfortable with our dataset. Now, let's assume some random set of hyper-parameters (a,b). For simplicity's sake, we'll change only a here keeping b as zero.

a = 0

b = 0

Our line equation y = ax+b now becomes

$$y = 0$$

Look at the black line now. It's equation is y=0 subjected to hyper-parameters (a=0,b=0)

Of course, we are using wrong a,b values as a demonstration, so this line is making its own different set of predictions (Green dots on the black line)

Anyhow, now our predictions are different from actual values, so let's calculate error using our cost function MSE.

$$error = \frac{1}{2*3} ((pred0 - actual0)^2 + (pred1 - actual1)^2 + (pred2 - actual2)^2)$$

$$error(0, 0) = \frac{1}{2*3} ((0 - 1000)^2 + (0 - 2000)^2 + (0 - 3000)^2 )$$

$$error(0, 0) = \frac{1}{6} (1000,000 + 4000,000 + 9000,000)$$

$$error(0, 0) = \frac{1}{6} * 14000,000$$

$$error(0, 0) = 2,333,333$$

Second set of hyper-parameters a=2000, b=0

Line equation: $$y=2000x$$

Look at the black line now. It's equation is y=2000x subjected to hyper-parameters (a=2000,b=0)

Let's calculate error using our cost function MSE.

$$error = \frac{1}{2*3} ((pred0 - actual0)^2 + (pred1 - actual1)^2 + (pred2 - actual2)^2)$$

$$error(2000, 0) = \frac{1}{2*3} ((2000 - 1000)^2 + (4000 - 2000)^2 + (6000 - 3000)^2 )$$

$$error(2000, 0) = \frac{1}{6} (1000,000 + 4000,000 + 9000,000)$$

$$error(2000, 0) = \frac{1}{6} * 14000,000$$

$$error(2000, 0) = 2,333,333$$

Third set of hyper-parameters a=500, b=0

Line equation: $$y=500x$$

Look at the black line now. It's equation is y=500x subjected to hyper-parameters (a=500,b=0)

Let's calculate error using our cost function MSE.

$$error = \frac{1}{2*3} ((pred0 - actual0)^2 + (pred1 - actual1)^2 + (pred2 - actual2)^2)$$

$$error(500, 0) = \frac{1}{2*3} ((500 - 1000)^2 + (1000 - 2000)^2 + (1500 - 3000)^2 )$$

$$error(500, 0) = \frac{1}{6} (500^2 + 1000^2 + 1500^2)$$

$$error(500, 0) = \frac{1}{6} (250,000 + 1000,000 + 2250,000)$$

$$error(500, 0) = \frac{1}{6} * 3500,000$$

$$error(500, 0) = 583,333$$

Fourth set of hyper-parameters a=1500, b=0

Line equation: $$y=1500x$$

Look at the black line now. It's equation is y=1500x subjected to hyper-parameters (a=1500,b=0)

Let's calculate error using our cost function MSE.

$$error = \frac{1}{2*3} ((pred0 - actual0)^2 + (pred1 - actual1)^2 + (pred2 - actual2)^2)$$

$$error(1500, 0) = \frac{1}{2*3} ((1500 - 1000)^2 + (3000 - 2000)^2 + (4500 - 3000)^2 )$$

$$error(1500, 0) = \frac{1}{6} (500^2 + 1000^2 + 1500^2)$$

$$error(1500, 0) = \frac{1}{6} (250,000 + 1000,000 + 2250,000)$$

$$error(1500, 0) = \frac{1}{6} * 3500,000$$

$$error(1500, 0) = 583,333$$

Fifth set of hyper-parameters a=250, b=0

Line equation now becomes: $$y=250x$$

Look at the black line now. It's equation is y=250x subjected to hyper-parameters (a=250,b=0)

Let's calculate error using our cost function MSE.

$$error = \frac{1}{2*3} ((pred0 - actual0)^2 + (pred1 - actual1)^2 + (pred2 - actual2)^2)$$

$$error(250, 0) = \frac{1}{2*3} ((250 - 1000)^2 + (500 - 2000)^2 + (750 - 3000)^2 )$$

$$error(250, 0) = \frac{1}{6} (750^2 + 1500^2 + 2250^2)$$

$$error(250, 0) = \frac{1}{6} (562,500 + 2250,000 + 5062,500)$$

$$error(250, 0) = \frac{1}{6} * 7875,000$$

$$error(250, 0) = 1312,500$$

Look at the plot above which contains hyper-parameter (a) values on x axis and respective Mean Squared Error on y axis. You could notice the behaviour of MSE that as the hyper-parameter reaches the optimum value (a=1000), MSE is minimised.

In this whole mathematical approach, we tried out bunch of hyper-parameter combinations till we finally found the optimum one. There's a very useful algorithm called gradient descent, let's for simplicity sake, say that gradient descent works the same way.

First, the cost function is calculated with random set of hyper-parameters, then step by step that cost function (MSE in this case) is reduced till optimum solution is found. In python, we usually use scikit-learn (sklearn) to do our machine learning tasks and this background working of gradient descent is usually hidden from the end user, but it helps to know such background algorithms to know how things work. We'll later talk in detail more about why we should learn gradient descent and how it helps.

Scikit-learn

So far, we've understood the mathematical side of Regression. Now, let's look at a very useful library of python which can do all these mathematical steps in usually 3 line.

We will pass x value of 4 (4 years of work experience) and generate prediction of salary

Salary prediction here was 4000