Linear Independence: Linear Algebra Study Notes

📐 Linear Independence

Main Topics Covered

• Definition of linear independence and linear dependence
• Independence of sequences, finite sets, infinite sets and indexed families
• Definition via span
• Geometric and geographic examples
• Evaluating linear independence: zero vector, two vectors, row reduction, determinants
• Natural basis vectors and linear independence of functions
• The space of linear dependencies
• Generalizations: affine independence and independent subspaces

💡 Core Idea

A set of vectors is linearly independent if there exists no vector in the set that is equal to a linear combination of the other vectors in the set. If such a vector exists, the vectors are linearly dependent.

• Linear independence is part of the definition of a linear basis.
• A vector space can be of finite or infinite dimension depending on the maximum number of linearly independent vectors.
• The definition of linear dependence, and the ability to decide whether a subset of a vector space is linearly dependent, are central to determining the dimension of a vector space.

📖 Definition

Linear dependence

A sequence of vectors $\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_k$ from a vector space $V$ is linearly dependent if there exist scalars $a_1, a_2, \dots, a_k$, not all zero, such that

$a_1\mathbf{v}_1 + a_2\mathbf{v}_2 + \cdots + a_k\mathbf{v}_k = \mathbf{0},$

where $\mathbf{0}$ denotes the zero vector.

• If $k = 1$: a single vector is linearly dependent if and only if it is the zero vector.
• If $k > 1$: at least one scalar is nonzero, say $a_1 \neq 0$, and the equation can be rewritten as

$\mathbf{v}_1 = \frac{-a_2}{a_1}\mathbf{v}_2 + \cdots + \frac{-a_k}{a_1}\mathbf{v}_k.$

• Therefore, a set of vectors is linearly dependent if and only if one of them is zero or a linear combination of the others.

Linear independence

A sequence $\mathbf{v}_1, \dots, \mathbf{v}_n$ is linearly independent if it is not linearly dependent, that is, if the equation

$a_1\mathbf{v}_1 + a_2\mathbf{v}_2 + \cdots + a_n\mathbf{v}_n = \mathbf{0}$

can only be satisfied by $a_i = 0$ for $i = 1, \dots, n$.

Equivalent ways to say it:

• No vector in the sequence can be represented as a linear combination of the remaining vectors.
• The only representation of $\mathbf{0}$ as a linear combination of the vectors is the trivial representation, in which all scalars $a_i$ are zero.
• Even more concisely: the vectors are linearly independent if and only if $\mathbf{0}$ can be represented as a linear combination of them in a unique way.

Sequences versus sets

• If a sequence contains the same vector twice, it is necessarily dependent.
• Linear dependency of a sequence does not depend on the order of its terms. This allows defining independence for a finite set: a finite set is linearly independent if the sequence obtained by ordering it is linearly independent.
• Useful result: a sequence of vectors is linearly independent if and only if it does not contain the same vector twice and the set of its vectors is linearly independent.

Infinite case

• An infinite set of vectors is linearly independent if every finite subset is linearly independent.
  • This also applies to finite sets, since a finite set is a finite subset of itself.
  • Every subset of a linearly independent set is also linearly independent.
• Conversely, an infinite set is linearly dependent if it contains a finite subset that is linearly dependent, or equivalently, if some vector in the set is a linear combination of other vectors in the set.
• An indexed family of vectors is linearly independent if it does not contain the same vector twice and the set of its vectors is linearly independent. Otherwise it is linearly dependent.
• A set that is linearly independent and spans a vector space forms a basis for that space.
  • Example: the vector space of all polynomials in $x$ over the reals has the (infinite) subset $\{1, x, x^2, \dots\}$ as a basis.

Definition via span

Let $V$ be a vector space.

• A set $X \subseteq V$ is linearly independent if and only if $X$ is a minimal element of $\{Y \subseteq V \mid X \subseteq \operatorname{Span}(Y)\}$ under the inclusion order.
• In contrast, $X$ is linearly dependent if it has a proper subset whose span is a superset of $X$.

🧭 Examples

Geometric examples

Vectors	Status	Reason
$\vec{u}$, $\vec{v}$	Independent	They define the plane $P$
$\vec{u}$, $\vec{v}$, $\vec{w}$	Dependent	All three lie in the same plane
$\vec{u}$, $\vec{j}$	Dependent	They are parallel to each other
$\vec{u}$, $\vec{v}$, $\vec{k}$	Independent	$\vec{u}$, $\vec{v}$ are independent and $\vec{k}$ is not a linear combination of them (they do not share a common plane); the three define a three-dimensional space
$\vec{o}$ (null vector), $\vec{k}$	Dependent	$\vec{o} = 0\,\vec{k}$

Geographic location

A person says, "It is 3 miles north and 4 miles east of here." This is enough to describe the location, since the geographic coordinate system may be considered a 2-dimensional vector space (ignoring altitude and the curvature of the Earth).

• The "3 miles north" and "4 miles east" vectors are linearly independent: the north vector cannot be described in terms of the east vector, and vice versa.
• Adding "5 miles northeast of here" is true but unnecessary. This third vector is a linear combination of the other two, so the set becomes linearly dependent: one of the three vectors is unnecessary.
• If altitude is not ignored, a third vector must be added to the independent set.
• In general, $n$ linearly independent vectors are required to describe all locations in $n$-dimensional space.

🔍 Evaluating Linear Independence

The zero vector

If one or more vectors in $\mathbf{v}_1, \dots, \mathbf{v}_k$ is the zero vector, the vectors are necessarily linearly dependent.

Why: suppose $\mathbf{v}_i = \mathbf{0}$.

1. Let $a_i := 1$ (any other nonzero scalar also works).
2. Let $a_j := 0$ for every index $j \neq i$, so that $a_j\mathbf{v}_j = 0\,\mathbf{v}_j = \mathbf{0}$.
3. Then

$a_1\mathbf{v}_1 + \cdots + a_k\mathbf{v}_k = \mathbf{0} + \cdots + \mathbf{0} + a_i\mathbf{v}_i + \mathbf{0} + \cdots + \mathbf{0} = a_i\mathbf{v}_i = a_i\mathbf{0} = \mathbf{0}.$

4. Since $a_i \neq 0$, not all scalars are zero, so the vectors are linearly dependent.

Consequences:

• The zero vector cannot belong to any linearly independent collection.
• For $k = 1$: the sequence $\mathbf{v}_1$ is linearly dependent if and only if $\mathbf{v}_1 = \mathbf{0}$; it is linearly independent if and only if $\mathbf{v}_1 \neq \mathbf{0}$.

Two vectors

For two vectors $\mathbf{u}$ and $\mathbf{v}$ from a real or complex vector space, they are linearly dependent if and only if at least one of these holds:

1. $\mathbf{u} = c\mathbf{v}$ for some scalar $c$, or
2. $\mathbf{v} = c\mathbf{u}$ for some scalar $c$.

Details of the cases:

• If $\mathbf{u} = \mathbf{0}$, take $c := 0$: $c\mathbf{v} = 0\mathbf{v} = \mathbf{0} = \mathbf{u}$, so (1) is true. Similarly, if $\mathbf{v} = \mathbf{0}$ then (2) is true because $\mathbf{v} = 0\mathbf{u}$.
• If $\mathbf{u} = \mathbf{v}$ (for instance both zero), both (1) and (2) are true (using $c := 1$).
• If $\mathbf{u} = c\mathbf{v}$ and $\mathbf{u} \neq \mathbf{0}$, then $c \neq 0$ and $\mathbf{v} \neq \mathbf{0}$, so multiplying both sides by $\frac{1}{c}$ gives $\mathbf{v} = \frac{1}{c}\mathbf{u}$.
• Hence if $\mathbf{u} \neq \mathbf{0}$ and $\mathbf{v} \neq \mathbf{0}$, then (1) is true if and only if (2) is true: either both are true (dependent) or both are false (independent).
• If exactly one of $\mathbf{u}$, $\mathbf{v}$ is $\mathbf{0}$ (the other nonzero), then exactly one of (1) and (2) is true.

Result: $\mathbf{u}$ and $\mathbf{v}$ are linearly independent if and only if $\mathbf{u}$ is not a scalar multiple of $\mathbf{v}$ and $\mathbf{v}$ is not a scalar multiple of $\mathbf{u}$.

Vectors in $\mathbb{R}^2$ (row reduction)

Three vectors: $\mathbf{v}_1 = (1,1)$, $\mathbf{v}_2 = (-3,2)$, $\mathbf{v}_3 = (2,4)$. Dependence requires nonzero scalars with

$a_1\begin{bmatrix}1\\1\end{bmatrix} + a_2\begin{bmatrix}-3\\2\end{bmatrix} + a_3\begin{bmatrix}2\\4\end{bmatrix} = \begin{bmatrix}0\\0\end{bmatrix},$

or equivalently

$\begin{bmatrix}1&-3&2\\1&2&4\end{bmatrix}\begin{bmatrix}a_1\\a_2\\a_3\end{bmatrix} = \begin{bmatrix}0\\0\end{bmatrix}.$

• Subtract the first row from the second:

$\begin{bmatrix}1&-3&2\\0&5&2\end{bmatrix}\begin{bmatrix}a_1\\a_2\\a_3\end{bmatrix} = \begin{bmatrix}0\\0\end{bmatrix}.$

• Divide the second row by 5, then multiply by 3 and add to the first row:

$\begin{bmatrix}1&0&16/5\\0&1&2/5\end{bmatrix}\begin{bmatrix}a_1\\a_2\\a_3\end{bmatrix} = \begin{bmatrix}0\\0\end{bmatrix}.$

• Rearranging:

$\begin{bmatrix}a_1\\a_2\end{bmatrix} = -a_3\begin{bmatrix}16/5\\2/5\end{bmatrix}.$

Nonzero $a_i$ exist, so $\mathbf{v}_3$ can be written in terms of $\mathbf{v}_1$ and $\mathbf{v}_2$: the three vectors are linearly dependent.

Two vectors: $\mathbf{v}_1 = (1,1)$ and $\mathbf{v}_2 = (-3,2)$ give

$\begin{bmatrix}1&-3\\1&2\end{bmatrix}\begin{bmatrix}a_1\\a_2\end{bmatrix} = \begin{bmatrix}0\\0\end{bmatrix}.$

The same row reduction yields

$\begin{bmatrix}1&0\\0&1\end{bmatrix}\begin{bmatrix}a_1\\a_2\end{bmatrix} = \begin{bmatrix}0\\0\end{bmatrix},$

so $a_i = 0$ and the vectors are linearly independent.

Vectors in $\mathbb{R}^4$

Are the three vectors

$\mathbf{v}_1 = \begin{bmatrix}1\\4\\2\\-3\end{bmatrix},\quad \mathbf{v}_2 = \begin{bmatrix}7\\10\\-4\\-1\end{bmatrix},\quad \mathbf{v}_3 = \begin{bmatrix}-2\\1\\5\\-4\end{bmatrix}$

linearly dependent? Form the matrix equation

$\begin{bmatrix}1&7&-2\\4&10&1\\2&-4&5\\-3&-1&-4\end{bmatrix}\begin{bmatrix}a_1\\a_2\\a_3\end{bmatrix} = \begin{bmatrix}0\\0\\0\\0\end{bmatrix}.$

Row reduction gives

$\begin{bmatrix}1&7&-2\\0&-18&9\\0&0&0\\0&0&0\end{bmatrix}\begin{bmatrix}a_1\\a_2\\a_3\end{bmatrix} = \begin{bmatrix}0\\0\\0\\0\end{bmatrix}.$

Rearranging to solve for $\mathbf{v}_3$:

$\begin{bmatrix}1&7\\0&-18\end{bmatrix}\begin{bmatrix}a_1\\a_2\end{bmatrix} = -a_3\begin{bmatrix}-2\\9\end{bmatrix}.$

This is solved by $a_1 = -3a_3/2$ and $a_2 = a_3/2$, where $a_3$ can be chosen arbitrarily. Nonzero $a_i$ exist, so $\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3$ are linearly dependent.

Alternative method using determinants

$n$ vectors in $\mathbb{R}^n$ are linearly independent if and only if the determinant of the matrix formed by taking the vectors as columns is non-zero.

Example: for $(1,1)$ and $(-3,2)$,

$A = \begin{bmatrix}1&-3\\1&2\end{bmatrix},\qquad A\Lambda = \begin{bmatrix}1&-3\\1&2\end{bmatrix}\begin{bmatrix}\lambda_1\\\lambda_2\end{bmatrix}.$

We ask whether $A\Lambda = 0$ for some nonzero $\Lambda$. This depends on the determinant:

$\det A = 1\cdot 2 - 1\cdot(-3) = 5 \neq 0.$

The determinant is non-zero, so the vectors are linearly independent.

Fewer vectors than coordinates ($m < n$):

• Suppose there are $m$ vectors with $n$ coordinates, $m < n$. Then $A$ is an $n \times m$ matrix, $\Lambda$ is a column vector with $m$ entries, and $A\Lambda = 0$ is a list of $n$ equations.
• Any solution of the full list must also solve the reduced list formed from any $m$ rows $\langle i_1, \dots, i_m\rangle$:

$A_{\langle i_1,\dots,i_m\rangle}\Lambda = \mathbf{0}.$

• The reverse is also true: the $m$ vectors are linearly dependent if and only if $\det A_{\langle i_1,\dots,i_m\rangle} = 0$ for all possible lists of $m$ rows.
• If $m = n$, only one determinant is needed. If $m > n$, it is a theorem that the vectors must be linearly dependent.
• This is valuable for theory; in practical calculations more efficient methods are available.

More vectors than dimensions

If there are more vectors than dimensions, the vectors are linearly dependent. This was illustrated by the three vectors in $\mathbb{R}^2$ above.

🧱 Natural Basis Vectors

Let $V = \mathbb{R}^n$ and consider the natural basis vectors

$\begin{aligned}\mathbf{e}_1 &= (1,0,0,\ldots,0)\\ \mathbf{e}_2 &= (0,1,0,\ldots,0)\\ &\;\;\vdots\\ \mathbf{e}_n &= (0,0,0,\ldots,1).\end{aligned}$

Then $\mathbf{e}_1, \mathbf{e}_2, \ldots, \mathbf{e}_n$ are linearly independent.

📈 Linear Independence of Functions

Let $V$ be the vector space of all differentiable functions of a real variable $t$. Then the functions $e^t$ and $e^{2t}$ in $V$ are linearly independent.

Proof

1. Suppose $a$ and $b$ are real numbers such that $ae^t + be^{2t} = 0$ for all $t$.
2. Take the first derivative: $ae^t + 2be^{2t} = 0$.
3. We need to show $a = 0$ and $b = 0$. Subtract the first equation from the second: $be^{2t} = 0$.
4. Since $e^{2t}$ is not zero for some $t$, $b = 0$.
5. It follows that $a = 0$ too.
6. By the definition of linear independence, $e^t$ and $e^{2t}$ are linearly independent.

🗂️ Space of Linear Dependencies

• A linear dependency (or linear relation) among vectors $\mathbf{v}_1, \dots, \mathbf{v}_n$ is a tuple $(a_1, \dots, a_n)$ of $n$ scalars such that

$a_1\mathbf{v}_1 + \cdots + a_n\mathbf{v}_n = \mathbf{0}.$

• If such a relation exists with at least one nonzero component, the $n$ vectors are linearly dependent.
• Linear dependencies among $\mathbf{v}_1, \dots, \mathbf{v}_n$ form a vector space.
• If the vectors are given by coordinates, the linear dependencies are the solutions of a homogeneous system of linear equations whose coefficients are the coordinates of the vectors.
• A basis of the space of linear dependencies can therefore be computed by Gaussian elimination.

🌐 Generalizations

Affine independence

• A set of vectors is affinely dependent if at least one vector can be defined as an affine combination of the others; otherwise it is affinely independent.
• Any affine combination is a linear combination, so every affinely dependent set is linearly dependent. Contrapositively, every linearly independent set is affinely independent.
• An affinely independent set is not necessarily linearly independent.
• Test: take $m$ vectors $\mathbf{v}_1, \ldots, \mathbf{v}_m$ with $n$ components each, and form $m$ augmented vectors with $n+1$ components each, with a leading $1$ placed on top of each vector. The original vectors are affinely independent if and only if the augmented vectors are linearly independent.

Linearly independent vector subspaces

• Two subspaces $M$ and $N$ of a vector space $X$ are linearly independent if $M \cap N = \{0\}$.
• More generally, subspaces $M_1, \ldots, M_d$ of $X$ are linearly independent if, for every index $i$,

$M_i \cap \sum_{k\neq i} M_k = \{0\},$

where $\sum_{k\neq i} M_k$ is the set of sums $m_1 + \cdots + m_{i-1} + m_{i+1} + \cdots + m_d$ with $m_k \in M_k$, which equals the span of the union of the $M_k$ for $k \neq i$.

• $X$ is the direct sum of $M_1, \ldots, M_d$ if these subspaces are linearly independent and $M_1 + \cdots + M_d = X$.