Geometry of linear equations for machine learning

A guide to help you understand linear algebra to get started with machine learning

The fundamental problem of linear algebra is to solve a system of equations. Those equations are linear, which means that the unknowns are only multiplied by numbers — we never see x² or x times y.

We will consider the case of ‘n’ linear equations, ‘n’ unknowns. This is the best and easy scenario where there are equal no. of linear equations and unknowns.

We will solve the equations using the below three methods:

1. Matrix form
2. Row picture
3. Column picture

1) Matrix form

Take 2 equations as shown below:

2x -y =0

x -2y =3

so the matrix form is:

Matrix (Source: Author)

AX=B

where,

A is coefficient matrix, X is the vector of unknown, B is a vector

2) Row picture

In this case, we begin one row at a time and plot it. The row picture shows two lines meeting at a single point (the solution).

Take 2 equations as shown below:

2x -y =0

x -2y =3

So take both the equations and plot the line. The plot can be seen below along with the point of intersection.

Plotted equations (Source: Author)

The two lines meet at x =1 and y = 2 and this point solves both equations.

3) Column picture

In this case, we will look at the columns and write the equation.

Take 2 equations as shown below:

x -2y =1

2x +y =7

If you separate the original system into its columns instead of its rows, you get a vector equation

Matrix (Source: Author)

this is called a linear combination of columns.

So here we have to find the perfect values of x and y in such a way that the equation is satisfied. The column picture combines the column vectors on the left side of the equations to produce the vector on the right side. With the right choice x=3 and y=1 we get the vector on the right.

Column picture (Source: MIT)

The column picture is more easier and preferred because in the row pictures as and when the no. of equations increase along with the no. of unknowns, the plotting gets harder to visualize and gets more complicated. For example if there are 3 equations and 3 unknowns. Then we will have to plot 3 planes and determine where it meets which can be a hassle.

Source

Now the question is if we can solve AX=B for every B? or

Do the linear combinations of columns fill the 3-D space?

The answer depends on the equations. Now if we consider 3 equations and 3 unknowns where all the column vectors lie in the same plane then the answer is NO. This is because their combinations will lie in the same plane and in this case the matrix will be not invertible and will be a singular case. There will be no solution.

Reach out to me: LinkedIn

Check out my other work: GitHub

Reference:

MIT linear algebra course