Definition
 

Inner Product and Inner Product Spaces

Norm defined by an inner product

 Example

0

Motivation

So far, we've seen that any two vectors can be naturally added and also multiplied by any scalar. But what if we want to define another structure for the vectors that gives a multiplicative result? The two vectors can be multiplied and this can be done by computing the dot product of the vectors. 
A dot product of two vectors $\vec{u} = (u_1, u_2, . . . . . . . , u_n)$ and $\vec{v} = (v_1, v_2, . . . . . . ., v_n)$ is formulated as $\vec{u}\cdot\vec{v}$ = $| \vec{u} |$ $| \vec{v} | $cos$\theta$,
where $| \vec{u} |$, $| \vec{v} |$ are the lengths of $\vec{u}$, $\vec{v} \in \mathbb{R}^n$ and $\theta$ is the angle between them. It is therefore possible to define the concepts of 'length' of the vectors and the 'angle' between them. 

Equivalently, the dot product can also be defined as 
$\vec{u}\cdot\vec{v} = u_1v_1 + u_2v_2 + . . . . . . . . + u_nv_n$

Dot product is used to find the angle between the two vectors.
Since, $\vec{u}\cdot\vec{v}$ = $| \vec{u} |$ $| \vec{v} | $cos$\theta$

$\implies$ cos$\theta$ = $\dfrac{\vec{u}\cdot\vec{v}}{| \vec{u} | | \vec{v} |}$

Now suppose you want to calculate the work done in moving an object from one point to another, then we know that the work done is the product of force applied and the distance between two points ($W = Fd$) but this is only correct when the force is applied in the same direction in which the object is moving. 
So when the force is applied in a different direction, then the work done will be the product of the distance and the magnitude of the component of the force that is applied in the direction of the moving object i.e |$F$| cos$\theta$. 
(For a better understanding on how to find components of a vector refer further reading [4])

Therefore, $W$ = |$d$| |$F$| cos$\theta$ which the dot product between the force and the displacement vector.

1

Bird's Eye View

One of the very important applications of the dot product is finding projections. Projection of any vector $\vec{u}$ is its shadow on another vector $\vec{v}$ and is defined as a component ($\vec{u}$cos$\theta$).

Mathematically, the projection of $\vec{u}$ onto $\vec{v}$ = $| \vec{v} |$cos$\theta$ = $ \dfrac{\vec{u}\cdot\vec{v}}{| \vec{u} |}$ where $\theta$ is the angle between $\vec{u}$ and $\vec{v}$.

3

3

The length of any vector $\vec{u}$ can be computed as: $| \vec{u} |$ = $\sqrt{\vec{u}\cdot\vec{u}}$.
However, this won't be valid if the vector we choose is in the complex coordinate plane. For example, if $\vec{u} = i$ then, $| \vec{u} |$ =  $\sqrt{i\cdot i}$ = $\sqrt{-1}$,
and as a consequence of this, the dot product of vectors in complex plane cannot be done.

For complex vectors, there is a generalized form of the dot product called Inner Product. The Inner Product of any two vectors $\vec{x}$ and $\vec{y}$ is defined as $\langle\vec{x}, \vec{y}\rangle$ = $\overline{\vec{x}}\cdot \vec{x}$, where $\vec{x}$ is the conjugate of $\vec{x}$. If we take any complex vector $\vec{x}$ then the length $| \vec{x} |$ = $\sqrt{\langle\vec{x}, \vec{x}\rangle}$.

Note: The conjugate of any complex number $z = a + ib$ is $\overline{z} = a  - ib$.

Now we can consider the above example where $\vec{u} = i$, then $| \vec{u} |$ = $\sqrt{\langle\vec{x}, \vec{x}\rangle}$ = $\sqrt{\overline{ i }\cdot i}$ = $\sqrt{-i\cdot i}$ = 1

Now, all the vectors in a space whose Inner Product can be computed forms an Inner Product Space.

3

Context of the Definition

We often consider our vector spaces over a real field but here, we shall discuss the vector spaces over the complex field as well.
An Inner Product is like a function in which we can give vectors from a vector space as inputs and get a scalar as an output.
The scalar that we get belongs to the field.

3

3

Let $V$ be a vector space over a field $F$, then the Inner Product defined is as $ \langle , \rangle : V$ x $V$ $\rightarrow$ $F$ is an operation that satisfies the following axioms:
If $u, v, w \in$ $V$, then 
                1)  $ \langle u + v, w \rangle $ = $ \langle u, w \rangle $ + $ \langle v, w \rangle $
                2)  $ \langle cu, v \langle $ = $c\langle u, v \rangle $
The first and second axiom talks about the 'Linearity' in the first variable. It will be satisfied whether the field $F$ is real or complex.

3

4

 3)  <$\overline{u, v}$> = <$v, u$>, where <$\overline{u, v}$> represents the complex conjugate.
The third axiom is about the  'Conjugate Symmetry' and this is also valid for both real and complex fields.
Let $F$ = $\mathbb{R}$, then the complex conjugate <$\overline{u, v}$> = <$u, v$> and if  $F$ = $\mathbb{C}$, then the complex conjugate <$\overline{u, v}$> = <$v, u$>.
If you remember the conjugate of a complex number $(x + iy)$ is $(x - iy)$. We can consider real numbers to be complex when its imaginary part is 0 i.e it can be written as x + 0$i$ and therefore, the complex conjugate of a real number is the number itself.

             4)  <$u, u$> $\geq$ 0 and <$u, u$> = 0 if and only if $u$ = 0.
The last axiom is about 'Positivity' of the Inner Product i.e. if $u$ is a non-zero vector then the inner product <$u, u$> will always be positive.

5

8

Lastly, the set ($V$, < , >) is called as Inner product space, where the Inner Product of two vectors is <$u, v$> = $\sum_{i=1}^{n} u_i v_i$.

Now let's look at some examples of the Inner product spaces.

Example 1: Let $V$ = $\mathbb{R}^n$ and $u, v \in$ $V$ such that $u = \{u_1, u_2, . . . . . . . . u_n\}$ and $v = \{v_1, v_2, . . . . . . . v_n\}$
Suppose $<u, v> = u\cdot v$, where $u\cdot v = u_1v_1 + u_2v_2 + . . . . . . . .u_nv_n$
This is the dot product for $\mathbb{R}^n$. 
So now we will check if it forms an Inner product space by verifying all of the defined axioms.

1)  Linearity
     Let $u, v, w \in \mathbb{R}^n$
     $<u + v, w> = (u +v)\cdot w = u\cdot w + v\cdot w = <u, w> + <v, w>$
2)  For a given $c$ in the field
     $<cu, v> = (cu)\cdot v = c(u\cdot v) = c<u, v>$
3)  Conjugate Symmetry
     By the similar argument that a real number's conjugate will be the number itself, we can say that
     <$\overline{u, v}> = \overline{u\cdot v} = u\cdot v =<v, u>$
4)  Let $u \neq$ 0, then $<u, u> = u\cdot u = u^2 >$ 0

Hence, $V$ = $\mathbb{R}^n$ forms an Inner Product space.

Example 2: Let $V = C [0, 2]$ be a vector space of all the continuous functions defined over real numbers. 
Now, $f(x), g(x) \in$ $C [0, 2]$ and the Inner Product $<f(x), g(x)> = \int_{0}^{2} f(x) \overline{g(x)} dx$
Since, $g(x) \in \mathbb{R} \implies \overline{g(x)} = g(x)$
$\therefore <f(x), g(x)> = \int_{0}^{2} f(x) g(x) dx$
1)  Let $f(x), g(x), h(x) \in$ $V$, then $<f(x) + g(x), h(x)> = \int_{0}^{2} ( f(x) + g(x) )h(x) dx = \int_{0}^{2} f(x)h(x) + g(x)h(x) dx$
= $\int_{0}^{2} f(x)h(x) + \int_{0}^{2} g(x)h(x)$ = $<f(x), h(x)> + <g(x), h(x)>$
2)  $<cf(x), g(x)> = \int_{0}^{2} cf(x)g(x) dx = c\int_{0}^{2} f(x)g(x) dx = c<f(x), g(x)$
3)  The conjugate of a real number is the number itself.
$<\overline{f(x), g(x)}> = <f(x), g(x)>$
4)  For a given $f(x) \neq$ 0, $<f(x), f(x)> = \int_{0}^{2} [f(x)]^2 dx > 0$

Hence, $V = C [0, 2]$ forms an Inner Product space.

Note: The following animation uses the property of definite integrals, i.e., $\int_{a}^{b} f(x)$ $dx$ = Area under the curve.                   

9

9

Norm of a vector

Let V be a vector space over a field F, then the length of vector $v \in$ V will be the Norm (denoted as $|| v ||$), defined as
$|| v ||$ = $\sqrt{<v, v>}$ = $\sqrt{{v_1}^2 + {v_2}^2 + . . . . . . . . {v_n}^2} \implies || v ||^2 = <v, v>$.

Suppose V = $\mathbb{R}^2$ and the vector $v \in$ V.
Consider $v = (v_1, v_2)$ and now if we visualize $v$ as a geometric point in a 2-dimensional plane, then we can say that the distance between the origin and point $(v_1, v_2)$ is $|| v ||$. This is a consequence of Pythagoras theorem used for triangles. 

10

10

Similarly, if we consider a cuboid then the length of its diagonal will be treated as the Norm of vector $v$ in $\mathbb{R}^3$.

For a given scalar $c \in$ F,
         $|| cv ||^2 = (cv)\cdot (cv) = c^2 v\cdot v = c^2 || v ||^2$
Taking square root on both sides,
         $|| cv || = | c |$ $|| v ||$

We've seen so far that an Inner Product is all about the length and angle between the two vectors whereas a Normed vector space helps in determining a metric on a space that measures the distance between the two vectors.

These spaces are ordered in such a way that, (Inner product spaces) $\subsetneq$ (Normed vector spaces) $\subsetneq$ (Metric spaces) $\subsetneq$ (Topological spaces)

The Norm of any vector $\vec{u}$ is $|| \vec{u} ||$ = $\sqrt{<\overline{u}, u>}$, we can say that the Inner Product is naturally associated with the Norm of vector and therefore, the Inner Product spaces can also be called a Normed vector spaces. 
Similarly, Normed vector space can also be called as a Metric space because it involves the idea of distance between any two vectors
$\vec{u}$ and $\vec{v}$ which is formulated in terms of norms of those vectors as  $|| u || - || v ||$. The elements of a metric space don't have to be vectors, they can be points as well. Lastly, if we consider a set of vectors or points in a metric space and discuss them having neighborhoods then the space will be called as a Topological space. The Inner Product spaces, Metric spaces, and Normed spaces are all instances of Topological spaces. Topological space is another interesting concept which will take us on a tangent, so for now let's stick to the Inner Product and Normed spaces only.

10

10

Applications

• The Inner Product is used by Electrical Engineers to compute the electric and magnetic flux. Flux is a measure of the electric/magnetic field through a given surface.

• The concept of the Inner Product is very useful in the field of Machine Learning.
  Suppose that you have three images, one of a dog, another of a cat, and the third one is of a lion. Now if you want a machine-learning algorithm to determine which one of the dog or cat is more likely to be used for labeling the lion then the algorithm goes through an abstract process that is similar to a dot product/ inner product in which all the three images will be treated as vectors. The dot product measures the angle between the lion and cat, and the dog and lion, then both the angles will be compared and classify the lion as a dog or cat according to how similar in direction cat/dog vector is to the lion vector.

• In Physics, Inner Product is used for calculating the work done. The work done is formulated as an Inner Product of the force applied and the displacement vector.

• The Inner Product is used to measure the size of the Earth and also its shape. It also measures the motion of the tides and poles and also the gravitational field to direct the actual coordinates of any point on the surface of the Earth.

11

History

Throughout the nineteenth century, from the time of Fourier and Dirichlet, Sturm and Liouville itself the notions of “Vector Space” and “Inner Product” were emerged. It was Fourier who arose the idea of inner product to determine the coefficient of the Fourier Series. While the wave equation was being solved for the vibration of a string given an initial displacement, it was observed that the sin and cos functions were well suited for it. Clairaut and Euler then found the integral orthogonality conditions, which permitted them to figure out the coefficients in the Fourier series expansion of the data. It was however viewed that these conditions were consistent only for the initial data, as only around 1760-1800, the concept that every function could be expanded arose. 
Later then, in 1825, the Cauchy-Schwarz inequality for Euclidean dot product was proven by Cauchy, but this was not linked with the orthogonal function expansion until later. Furthermore, in the years that followed, around the 1880s, the concept of norms, or integral distances as they called it was defined by Schwarz when he proved the Cauchy-Schwarz inequality for integrals. These norms would satisfy the triangle inequality and could be used to approach solutions of minimal surface. It was during this time itself when a real number was being defined meticulously for the first time. After this, the focus shifted on the solutions of Partial Differential Equations, including Fourier's, and the convergence of orthogonal expansions. Lebesgue's integral was also developed around this time itself with an expressed intent of studying the convergence of Fourier expansion. Even the modern notion of an operator was defined by Fredholm only when he was trying to find the solutions of the differential equations using integral equations. Later, during the 1900s, the axioms for $\ell^2$ were established by Hilbert, who is also considered to be the one to give the first definition of Inner Product space according to Dieudonne.

11

Pause and Ponder

1)  If we define an Inner Product of two vectors in a vector space V = $\mathbb{C}$ as a classic dot product i.e $<u,v> =       u\cdot v$,
     where $u, v \in$ V over a field of complex numbers then will it form an Inner Product space?

2)  Can you think of the possibilities where the Inner Product of the two vectors in a vector space be zero?

3)  Let V be a vector space over some field F and $u, v, w \in$ V. Now if $<u, v> = <u, w>$, then what can be said about 
     the relationship of the vectors $v$ and $w$?

11

References

[1]  Lay, David C. Linear algebra and its applications. Addison-Wesley,, 2012.  

[2]  Birkhoff, G., & Kreyszig, E. (1984). The establishment of functional analysis. Historia Mathematica, 11(3), 258-321.

[3]  levap (https://math.stackexchange.com/users/32262/levap), Relation between metric spaces, normed vector spaces, and inner product space., URL (version: 2018-07-05): https://math.stackexchange.com/q/2841873

[4]  Renze, John; Stover, Christopher; and Weisstein, Eric W. "Inner Product." From MathWorld--A Wolfram Web Resource. https://mathworld.wolfram.com/InnerProduct.html

[5]  http://www.princeton.edu/~aaa/Public/Teaching/ORF523/S17/ORF523_S17_Lec2_gh.pdf

[6]  https://prezi.com/p/2x9biebjhjwg/applications-of-inner-product-spaces/

[7] http://www.eng.fsu.edu/~dommelen/quantum/style_a/dot.html

12

Further Reading

[1]  https://sites.math.washington.edu/~greenbau/Math_555/Course_Notes/555notes5.ps_pages.pdf

[2]  https://www.math.usm.edu/lambers/mat610/sum10/lecture2.pdf

[3]  https://www.math.tamu.edu/~yvorobet/MATH304-2011A/Lect3-05web.pdf

[4] https://www.dummies.com/education/science/physics/how-to-find-vector-components/#:~:text=In%20physics%2C%20when%20you%20break,(vertical)%20component%20is%201.

13