Embedded Files
What is Linear Regression?
In statistics, linear regression is a linear approach to modelling the relationship between a dependent variable(y) and one or more independent variables(X). In linear regression, the relationships are modeled using linear predictor functions whose unknown model parameters are estimated from the data. Linear Regression is one of the most popular algorithms in Machine Learning. That’s due to its relative simplicity and well known properties.
Simple Linear Regression
Linear Regression is called simple if you are only working with one independent variable.
Formula: f(x)=mx+b
Cost Function
We can measure the accuracy of our linear regression algorithm using the mean squared error (mse) cost function. MSE measures the average squared distance between the predicted output and the actual output (label).
The implementation of MSE is pretty straight forward and we can easily code it up only using Python.
def cost_function(m, b, x, y):
totalError = 0
for i in range(0, len(x)):
totalError += (y[i]-(m*x[i]+b))**2
return totalError/float(len(x))
Optimization
To find the coefficients that minimize our error function we will use gradient descent. Gradient descent is a optimization algorithm which iteratively takes steps to the local minimum of the cost function.
To find the way towards the minimum we take the derivative of the error function in respect to our slope m and our y intercept b. Then we take a step in the negative direction of the derivative.
General Gradient Descent Formula
Gradient Descent for Simple Linear Regression
The implementation of gradient descent is a little bit more involved but it’s also easily doable in pure Python.
def gradient_descent(b, m, x, y, learning_rate, num_iterations):
N = float(len(x))
for j in range(num_iterations): # repeat for num_iterations
b_gradient = 0
m_gradient = 0
for i in range(0, len(x)):
b_gradient += -(2/N) * (y[i] - ((m * x[i]) + b))
m_gradient += -(2/N) * x[i] * (y[i] - ((m * x[i]) + b))
b -= (learning_rate * b_gradient)
m -= (learning_rate * m_gradient)
if j%50==0:
print('error:', cost_function(m, b, x, y))
return [b, m]
Running Linear Regression
In order to run our linear regression model we need to create a dataset and define a few initial variables. We will use pandas for creating simple random data and matplotlib to visualize it.
import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(0, 100, 50)
delta = np.random.uniform(-10, 10, x.size)
y = 0.5*x + 3 + delta
plt.scatter(x, y)
random data
Now that we have our data we are ready to train our model using the functions we defined above:
# defining some variables
learning_rate = 0.0001
initial_b = 0
initial_m = 0
num_iterations= 100
print('Initial error:', cost_function(initial_m, initial_b, x, y))
[b, m] = gradient_descent(initial_b, initial_m, x, y, learning_rate, num_iterations)
print('b:', b)
print('m:', m)
print('error:', cost_function(m, b, x, y))
Console Outputs
Now that we have trained our model we can use it to make predictions. In our simple example we will just predict on the training data and use the results to plot a best fit line using matplotlib.
predictions = [(m * x[i]) + b for i in range(len(x))]
plt.scatter(x, y)
plt.plot(x, predictions, color='r')
Best fit line
Google Sites
Report abuse