Linear Algebra Fundamentals

Linear Algebra is a branch of mathematics concerning vector spaces and linear mappings between these spaces. It includes the study of lines, planes, and subspaces, but is also fundamental in modern applied mathematics and engineering.

Vectors

A vector is an object that has both a magnitude and a direction. It is often represented as an ordered list of numbers (coordinates).

For example, a vector in 3D space can be written as:

\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ v_3 \end{bmatrix}

Vector Notation

Vectors are typically denoted in boldface, e.g., \mathbf{v}, or with an arrow overhead, e.g., \vec{v}.

Example

Vector \mathbf{a} = \begin{bmatrix} 3 \\ -2 \\ 5 \end{bmatrix} represents a vector in 3-dimensional space.

Vector Operations

Vector Addition

Vectors are added component-wise:

\mathbf{u} + \mathbf{v} = \begin{bmatrix} u_1 \\ u_2 \\ u_3 \end{bmatrix} + \begin{bmatrix} v_1 \\ v_2 \\ v_3 \end{bmatrix} = \begin{bmatrix} u_1 + v_1 \\ u_2 + v_2 \\ u_3 + v_3 \end{bmatrix}

Scalar Multiplication

Multiplying a vector by a scalar scales each component:

c \mathbf{v} = c \begin{bmatrix} v_1 \\ v_2 \\ v_3 \end{bmatrix} = \begin{bmatrix} c v_1 \\ c v_2 \\ c v_3 \end{bmatrix}

Dot Product

The dot product (or scalar product) of two vectors is a scalar defined as:

\mathbf{u} \cdot \mathbf{v} = u_1 v_1 + u_2 v_2 + u_3 v_3

The dot product is related to the angle between vectors:

\mathbf{u} \cdot \mathbf{v} = \|\mathbf{u}\| \|\mathbf{v}\| \cos \theta

Cross Product (3D only)

The cross product of two vectors in 3D results in a vector perpendicular to both:

\mathbf{u} \times \mathbf{v} = \begin{bmatrix} u_2 v_3 - u_3 v_2 \\ u_3 v_1 - u_1 v_3 \\ u_1 v_2 - u_2 v_1 \end{bmatrix}

Example: Dot Product

Given \mathbf{a} = \begin{bmatrix} 1 \\ 3 \\ -5 \end{bmatrix} and \mathbf{b} = \begin{bmatrix} 4 \\ -2 \\ -1 \end{bmatrix}, calculate \mathbf{a} \cdot \mathbf{b}.

Solution:

\(1 \times 4 + 3 \times (-2) + (-5) \times (-1) = 4 - 6 + 5 = 3\)

Matrices

A matrix is a rectangular array of numbers arranged in rows and columns.

A matrix with m rows and n columns is called an m \times n matrix.

Example of a 2 \times 3 matrix:

A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}

Matrix Notation

Capital letters like A, B, C denote matrices, with entries denoted by subscripts, e.g., a_{ij} is the element in row i and column j.

Matrix Operations

Matrix Addition

Two matrices of the same size can be added by adding their corresponding elements:

A + B = [a_{ij} + b_{ij}]

Scalar Multiplication

Multiplying a matrix by a scalar multiplies each element:

c A = [c a_{ij}]

Matrix Multiplication

The product of an m \times n matrix A and an n \times p matrix B is an m \times p matrix C, where:

c_{ij} = \sum_{k=1}^n a_{ik} b_{kj}

Example: Matrix Multiplication

Multiply matrices:

A = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}, B = \begin{bmatrix} 2 & 0 \\ 1 & 3 \end{bmatrix}

Solution:

Calculate AB:

\(c_{11} = 1 \times 2 + 2 \times 1 = 2 + 2 = 4\)

\(c_{12} = 1 \times 0 + 2 \times 3 = 0 + 6 = 6\)

\(c_{21} = 3 \times 2 + 4 \times 1 = 6 + 4 = 10\)

\(c_{22} = 3 \times 0 + 4 \times 3 = 0 + 12 = 12\)

Thus, AB = \begin{bmatrix} 4 & 6 \\ 10 & 12 \end{bmatrix}

Systems of Linear Equations

A system of linear equations consists of multiple linear equations involving the same variables.

Example:

\begin{cases} 2x + 3y = 5 \\ 4x - y = 11 \end{cases}

Systems can be solved by substitution, elimination, or matrix methods such as Gaussian elimination.

Gaussian Elimination

This method transforms the system's augmented matrix to row echelon form to solve variables step-by-step.

Example: Solve by Gaussian Elimination

Solve:

\begin{cases} x + 2y + z = 6 \\ 2x + 3y + 3z = 14 \\ x + y + 2z = 8 \end{cases}

Solution:

  1. Write augmented matrix:
    \begin{bmatrix} 1 & 2 & 1 & | & 6 \\ 2 & 3 & 3 & | & 14 \\ 1 & 1 & 2 & | & 8 \end{bmatrix}
  2. Eliminate below first row:
    Row 2 = Row 2 - 2 × Row 1:
    \begin{bmatrix} 1 & 2 & 1 & | & 6 \\ 0 & -1 & 1 & | & 2 \\ 1 & 1 & 2 & | & 8 \end{bmatrix}
  3. Eliminate below first row:
    Row 3 = Row 3 - Row 1:
    \begin{bmatrix} 1 & 2 & 1 & | & 6 \\ 0 & -1 & 1 & | & 2 \\ 0 & -1 & 1 & | & 2 \end{bmatrix}
  4. Eliminate below Row 2:
    Row 3 = Row 3 - Row 2:
    \begin{bmatrix} 1 & 2 & 1 & | & 6 \\ 0 & -1 & 1 & | & 2 \\ 0 & 0 & 0 & | & 0 \end{bmatrix}
  5. Solve back substitution:
    From Row 2: -y + z = 2 \Rightarrow y = z - 2
    From Row 1: x + 2y + z = 6
    Substitute y = z - 2:
    \(x + 2(z - 2) + z = 6 \Rightarrow x + 2z - 4 + z = 6 \Rightarrow x + 3z = 10 \Rightarrow x = 10 - 3z\)
  6. General solution: \(x = 10 - 3z, y = z - 2, z = z\) (parametric)

Determinants

The determinant is a scalar value that can be computed from the elements of a square matrix and encodes certain properties such as invertibility.

For a 2x2 matrix:

\det \begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc

For larger matrices, determinants can be calculated using cofactor expansion.

Example: Determinant of 3x3 matrix

Calculate determinant of:

A = \begin{bmatrix} 1 & 2 & 3 \\ 0 & 4 & 5 \\ 1 & 0 & 6 \end{bmatrix}

Solution:

Using cofactor expansion along first row:

\( \det A = 1 \times \det \begin{bmatrix} 4 & 5 \\ 0 & 6 \end{bmatrix} - 2 \times \det \begin{bmatrix} 0 & 5 \\ 1 & 6 \end{bmatrix} + 3 \times \det \begin{bmatrix} 0 & 4 \\ 1 & 0 \end{bmatrix} \)

= \(1 \times (4 \times 6 - 0 \times 5) - 2 \times (0 \times 6 - 1 \times 5) + 3 \times (0 \times 0 - 1 \times 4)\)

= \(1 \times 24 - 2 \times (-5) + 3 \times (-4) = 24 + 10 - 12 = 22\)

Eigenvalues and Eigenvectors

Given a square matrix A, an eigenvector \mathbf{v} and eigenvalue \lambda satisfy:

A \mathbf{v} = \lambda \mathbf{v}

Eigenvalues are found by solving the characteristic equation:

\det(A - \lambda I) = 0

Example: Find Eigenvalues

Matrix:

A = \begin{bmatrix} 4 & 1 \\ 2 & 3 \end{bmatrix}

Solution:

Characteristic equation:

\det \begin{bmatrix} 4 - \lambda & 1 \\ 2 & 3 - \lambda \end{bmatrix} = 0

\((4 - \lambda)(3 - \lambda) - 2 \times 1 = 0\)

\(\lambda^2 - 7\lambda + 10 = 0\)

Solve quadratic:

\(\lambda = \frac{7 \pm \sqrt{49 - 40}}{2} = \frac{7 \pm 3}{2}\)

Eigenvalues: \(\lambda_1 = 5, \lambda_2 = 2\)

Vector Spaces

A vector space is a set of vectors along with two operations (vector addition and scalar multiplication) that satisfy eight axioms such as closure, associativity, and distributivity.

Examples of vector spaces include \mathbb{R}^n, the set of all polynomials, and function spaces.

Linear Transformations

A linear transformation T: V \to W between vector spaces V and W satisfies:

  • T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})
  • T(c \mathbf{v}) = c T(\mathbf{v})

Examples include rotations, reflections, and projections.

Diagonalization

A matrix A is diagonalizable if it can be written as:

A = PDP^{-1} where D is diagonal and P is invertible.

This helps simplify matrix powers and solving differential equations.

Applications of Linear Algebra

  • Computer Graphics
  • Engineering
  • Machine Learning and Data Science
  • Quantum Mechanics
  • Economics and Statistics

Practice Problems

Problem 1: Find the dot product of \mathbf{u} = \begin{bmatrix} 2 \\ -1 \\ 3 \end{bmatrix} and \mathbf{v} = \begin{bmatrix} 4 \\ 0 \\ -2 \end{bmatrix}.

Solution:

\(2 \times 4 + (-1) \times 0 + 3 \times (-2) = 8 + 0 - 6 = 2\)

Problem 2: Calculate the determinant of \begin{bmatrix} 3 & 1 \\ 2 & 4 \end{bmatrix} .

Solution:

\(3 \times 4 - 2 \times 1 = 12 - 2 = 10\)

Problem 3: Find the eigenvalues of \begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix} .

Solution:

The eigenvalues are the diagonal elements: \(2, 3\)