Linear Algebra

Vectors

3 ways of looking at it:

• Arrow pointing in space (length & direction)
• Ordered list of numbers (features)

Best way to visualize is to think of an arrow with tail at the origin, on a plane. List of numbers = coordinates.

x \\ y \\ z \end{bmatrix}$$ ### Ops Add: Add the components, tip to tail method Scalar multiply: Multiply by a scalar (just mult all components) ### Special Vectors **Basis Vectors** $\hat{i}$, the unit vector (length 1) in x direction $\hat{j}$, the unit vector (length 1) in y direction $\hat{k}$, the unit vector (length 1) in x direction The reason it is called basis vectors is because it literally defines our coordinate system. ### Dot Products For full intuition, it's best you understand the [[#Matrixes|Later Sections]]. The dot product is just multiplying each corresponding element and taking their sum. The geometric interpretation is that it is $\vec{a} \cdot \vec{b} = |\vec{a}||\vec{b}|\cos \theta$. We project the vectors onto the same line and find their product. So how does the algebraic derivation work? Really, its just collapsing everything down to ONE dimensions. And if we think about it as moving the basis vectors, we're essentially saying "collapse x onto the number line by doing some number times x" (and for all other dimensions). Really the dot product is just a matrix mult! Dot products are also tools to see generally "which direction" a vector points relative to another ### Cross Products Similar thing, try to gain full intuition. The cross product is: the area of the parallelogram formed by two vectors. For $\vec{v} \times \vec{w}$, if $\vec{v}$ is to the right of $\vec{w}$, it is positive, else, negative. The way you can easily solve is by finding the determinant. Say $\vec{v} = [3, 1]$ and $\vec{w} = [2, -1]$, then area is just $\det(\begin{bmatrix}3 & 1 \\ 1 & -1\end{bmatrix})$ However, the cross product is meant to be a 3D VECTOR,. With length of that AREA, perpendicular to the parallelogram (RHR). The way you're suppose to go about this is weird:

\det(\begin{bmatrix}
\hat{i} & v_{1} & w_{1} \\
\hat{j} & v_{2} & w_{2} \\
\hat{k} & v_{3} & w_{3} 
\end{bmatrix})
=
\hat{i}(v_{2}w_{3}-v_{3}w_{2}) + \hat{j}(v_{3}w_{1} - v_{_{1}w_{3}}) + \hat{k}(v_{1}w_{2} - v_{2}w_{1}) 

### Terminology **Span**: the set of all linear combination of $a \vec{v} + b \vec{w}$ **Linearly dependent**: one of the vectors is a linear combination of others (does not improve the span) **Linearly independent**: Adds to the span ## Matrixes Matrixes are linear TRANSFORMATIONS, not just random ops. Linear transformations work by moving the BASIS vector away from $\hat{i}$ and $\hat{j}$ . ### Matrix-Vector For example $$\begin{bmatrix}-1 \\ 2\end{bmatrix}$$ is 1 of the first basis vector + 2 of the second basis. We can move the basis vectors around so instead of being a line on x-axis, it can be a wacky diagonal line. Lets rotate that by transforming $\hat{i}$ to $\begin{bmatrix}1 \\ -2\end{bmatrix}$ to $\hat{j}$ to $\begin{bmatrix}3 \\ 0\end{bmatrix}$ Transformed v becomes $-1\begin{bmatrix}1 \\ -2\end{bmatrix} + 2\begin{bmatrix}3 \\ 0\end{bmatrix} = \begin{bmatrix}-1(1) + 2(3) \\ -1 (-2) + 2(0)\end{bmatrix} = \begin{bmatrix}5 \\ 2\end{bmatrix}$ We can pack this into a **matrix**:

\begin{bmatrix}
3 & 2 \\
-2 & 1
\end{bmatrix}

$$Soinamoregeneralway:$$

\begin{bmatrix}
a & b \\
c & d
\end{bmatrix}
\begin{bmatrix}
x \\
y
\end{bmatrix}
=
x
\begin{bmatrix}
a \\
c
\end{bmatrix}
+
y
\begin{bmatrix}
b \\
d
\end{bmatrix}
=
\begin{bmatrix}
ax + by \\
cx + dy
\end{bmatrix}

### Matrix-Matrix We can think of a matrix-matrix mult as composition of different transformations (ex. first rotate, then shear) So for example, we have this matrix that shears:

\begin{bmatrix}
1 & 1 \\
0 & 1
\end{bmatrix}

$$Andwehavethismatrixthatrots:$$

\begin{bmatrix}
0 & -1  \\
1 & 0
\end{bmatrix}

$$Soshould{n}^{\prime }t:$$

\begin{bmatrix}
1 & 1 \\
0 & 1
\end{bmatrix}
\begin{bmatrix}
0 & -1 \\
1 & 0 
\end{bmatrix}
=
\begin{bmatrix}
1 & -1 \\
1 & 0
\end{bmatrix}

$$\ast Readrighttoleft,itsfunctioncompositionf(g(x))ifyouthinkaboutit\ast Ifwereallythinkaboutit,w{e}^{\prime }retranslatingtherightvecto{r}^{\prime }si-hat:$$

\begin{bmatrix}
1 & 1 \\
0 & 1
\end{bmatrix}
\begin{bmatrix}
0 \\
1 \\
\end{bmatrix}
=
\begin{bmatrix}
1 \\
1
\end{bmatrix}

$$Andwearetranslatingtherightvectors,j-hat$$

\begin{bmatrix}
1 & 1 \\
0 & 1
\end{bmatrix}
\begin{bmatrix}
-1 \\
0
\end{bmatrix}
=
\begin{bmatrix}
-1 \\
0
\end{bmatrix}

**ORDER MATTERS** This can easily be extended to higher dimensions. ### The Determinant Some transformations stretch, others shrink. Sometimes it's important to find exactly how much the *area* is scaled. That is the **determinant**. When determinant is 0, it collapses to a smaller dimension. Determinants can be negative. That means it has flipped the area (think: flipping sheet of paper). In 3D, use the right hand rule to determine if the shape has been flipped. For a 2x2 matrix,

\det(
\begin{bmatrix}
a & b \\
c & d 
\end{bmatrix}
) 
= 
ad - bc

The proof involves drawing a large square of (a + b)(d + c) and then removing ac's and bd's in it. ### Solving Linear Equations Linear equations are essentially matrix ops! Given:

\begin{align}
2x + 5y +3z &= -3 \
4x + 0y + 8z &= 0 \
1x + 3y + 0z &= 2
\end{align}

$$Itisesentiallytheequation$$

\begin{bmatrix}
2 & 5 & 3 \\
4 & 0 & 8 \\
1 & 3 & 0
\end{bmatrix}
\begin{bmatrix}
x \\
y \\
z
\end{bmatrix}
=
\begin{bmatrix}
-3 \\
0 \\
2
\end{bmatrix}

If we think about it, we have some vector x, y, z, and we're transforming it to end up at -3, 0, 2. We call this matrix transform $A$. If we know the inverse $A^{-1}$, we can solve for (x, y, z). But if $\det(A) = 0$, then the equations are linearly dependent(they're the same equations! or totally incnsistent equations). That equation collapses a whole dimension, and you cannot reverse that. And because you collapsed a dimension, now many inputs can map to a same point on that line. Using [[#The Determinant]] we can actually solve this! **Important: First solution thinks we can use dot products, but DOT PRODUCTS no longer becomes same when LINEARLY TRANSFORMED** Heres the interesting through: What if we used determinants? The area of the parallelogram formed by (x, y) and the basis vector i, gives us the y coordinate. **Area stays the same when transformed**. And to find the area of the scaled parallelogram, we can use the determinant (remember [[#Cross Products]]? yea, we're just pretending that they are our linear transformations) And then, we just divide determinants! ### Non-square? You can definitely have weird vector sizes (for example, mapping 2d => 3d space!). Same logic of transforming basis vectors. # of columns = the input dimensions, # of rows = the output dimensions. ### Terminology **Linear transformation**: A function that manipulates position (movement). Linear = lines are not curved, origin is fixed in space **Shear** - keep on axis same, shift another **Identity matrix** - the "1" of the matrix world, leaves unit vectors alone **Rank** - number of dimensions in the OUTPUT of a matrix **Full Rank** - max numbers **Column space** - all possible outputs of matrix **Null space / kernel** - space of all vectors that when multiplied by matrix, become "null" (zero vector) **m x n** - matrix with m rows and n columns ## Vector spaces All vectors are on a basis, implicitly $\hat{i}$ and $\hat{j}$. But two different people can have two different "meanings". For example, our (1, 0) might be some other persons (2, 1) ### Eigenvectors and Values There are some special vectors where even after moving the basis vectors, its span remains the same. Meaning that even though its scaled, it still is one the same line. That is an Eigenvector, and it's Eigennumber is how much it is "scaled". Eigenvectors can be the "axis of rotation" Expressesed as: $A\vec{v} = \lambda \vec{v}$, where A is the transformation, $\vec{v}$ is the eigenvector and $\lambda$ is the eigenvalue.

\lambda I = \begin{bmatrix}
\lambda & 0 & 0 \
0 & \lambda & 0 \
0 & 0 & \lambda
\end{bmatrix}

So to find the Eigenvectors & values:

(A-\lambda I)\vec{v} = \vec{0}

And in order for a non-zero vector to be transformed into one, we need a collapsing of dimensions So $$\det(A-\lambda I) = \det(\begin{bmatrix} a - \lambda & b \\ c & d - \lambda \end{bmatrix}) = 0$$ One eigenvalue does NOT mean 1 eigen vector! Can be multiple