Linear Algebra and Linear models

In today's workshop, you will learn a tiny bit about the deep relationship that exists between machine learning and linear algrabra. In particular, we will focus on understanding how concepts from linear algebra can be used to obtain linear regression models.

Today's learning objectives

• Understand the relationship between linear algebrea and linear models.
• Familiarise yourself with the normal equations to solve linear models.

Linear Regression - a Linear Algebra perspective

Recall that while building a linear regression models, we generally have an outcome variable y, which we want to predict using a set of predictor variables X. For example, y could refer to the number of ice creams sold, and X could refer to the temperature.

In linear algebra notation, y is then a column vector of variables.

Y = [[y₁, y₂, y₃, …, y_N]]

And in the case of a simple linear regression, x is a column vector similar in shape to y.

x = [[x₁, x₂, x₃, …, x_N]]

In the case of multiple linear regression, X is now a matrix of predictor variables. Assuming we have N rows of K variables:

X = [[x_1,1, x_1,2, x_1,3, …, x_1,K], [x_2,1, x_2,2, x_2,3, …, x_2,K], …, [x_N,1, x_N,2, x_N,3, …, x_N,K]]

We cam now write down our linear regression model as

y = beta.X

Where:
beta is a vector of regression coefficients
error is the irreducible error

Recall from the previous lesson that linear systems can be represented in linear algebra form as:

Ax = b

Since we have now represented our linear regression model as a linear system, we can solve it using tools of linear algebra.

Solving linear regression - the Normal equations

The solution to the linear regression problem we stated as

y = beta.X

is given by

beta_hat = (X^TX)^-1.X^Ty

Proving the normal equations requires mathematical tools that will be introduced in Year 2. We will re-visit the normal equations then :).

Tomorrow in the data lab, we will implement the normal equations and learn how they work in practice.

Today's takeaway is that the normal equations are beautiful and a direct application of linear algebra concepts to machine learning!

Blended Learning - Code Academy OR Khan Academy

• Please complete with the Linear Algebra course in Code Academy which can be found here and upload a certificate of completion to your learning logs.

OR

• Please complete with the Matrices course in Khan Academy which can be found here and upload evidence of completion to your learning logs.