0.1 Linear Algebra

Linear algebra is a foundational building block of ML as majority of the components are written in very high dimensional arrays and tensors. Without exaggeration, training and using a model can be considered as performing billions of linear algebra operations. Therefore, it is encouraged to get familiar with the notations as they will be used throughout the future notes. That being said, because many of the contents were introduced in detail in the linear algebra part of LibreNotebook, I will keep it brief here for topics that were not introduced in the previous part.

0.1.1 Foundations

Many mathematicians might not like the definition below, but I think it is worth considering different perspectives of how we can see vectors and matrices.

Now scalars are not considered vectors because it is only defined with magnitude. Geometrically speaking, the dimensions here can be interpreted as the dimension that we usually say in science. For example, a matrix can be plotted in a 2D space with \(x\) and \(y\) components. Similarly, a 3D tensor can be drawn in 3-dimensional plane with \(x\), \(y\), and \(z\) coordinates. Consider the following example.

Continuing, we can see an operation that vectors can perform that we have not discussed in the notes for linear algebra.

Dot product will be quite important when we discuss attention and transformers because it can tell us how “related” two vectors are. Indeed, consider the vectors \(A\) and \(B\) with the angle between the two vectors as \(\theta \). The following equation holds. \[ A \cdot B = ||A|| ||B|| \cos \theta \] Here, \(||V||\) for \(V \in \mathbb {R}^n\) represents the magnitude of the vector, i.e. \(\sqrt {\sum _{k=1}^n V_k^2}\). Let’s take a look at a quick example. \begin{align*} \langle 1, 0 \rangle \cdot \langle 1, \sqrt {3} \rangle &= 1 \cdot 1 + 0 \cdot \sqrt {3} = 1 \\ &= 1 \cdot 2 \cdot \cos 60^\circ = 1 \end{align*}

One thing to keep in mind is that we have a special term for the sign \(||V||\).

With that in mind, let’s move on to a more abstract and advanced topics in linear algebra that are constantly used in machine learning.

0.1.2 CONTINUE