On the page 'Orthogonal, Orthonormal', I tried to illustrate the
importance of perpendicular vectors in a transform matrix. Without
that, a satisfying reversal of the transform would not be possible. On
this page we will check which sinusoids are perpendicular and which are
not.

Below, the product function sin(2*pi*x)*cos(2*pi*x) is
illustrated. The inner product could be computed; it is the sum of the
products over
the interval of one period. From the figure it can be visually
verified, that the sum will be zero, because the
negative parts will exactly cancel the positive parts. Vectors and
functions with inner product zero are orthogonal or perpendicular. Thus
we see that sin(2*pi*x) and cos(2*pi*x) are orthogonal.

From here on I will have to use different scales on the x and y
axis, otherwise the curves would be squeezed to the point of being
indistinguishable. Below is exactly the same representation once again
but with the x axis stretched.

Now I have space to double the periodicity of sine and cosine. As
can be seen from the product curve, this does not affect the
orthogonality. Sine and cosine of equal periodicity can be used as a
pair of correlation vectors, which together will cover all phases of
real signals with that periodicity.

But it turns out that the pair cos(2*pi*x)+sin(2*pi*x) is also
orthogonal to the pair cos(2*pi*x)-sin(2*pi*x). These pairs are
centered around zero.

Strictly, if you want a complete orthogonal matrix with sine/cosine
pairs, or complex vectors as they say, you have to include the vectors
with negative sine as well. These are called the complex conjugates.
You then have pairs of pairs, which are called complex
conjugate pairs.

At first sight, this may seem to complicate matters. But that is not
quite the case. With the complex conjugates included, matrices become
symmetrical and invertible. Formulas become simpler. Moreover, such
complete complex matrices open up the way to
matrix factorization and very effective algorithms, like the Fast
Fourier Transform. So let us give the conjugates a place in our heart.

Till here I have illustrated integrals of sinusoids with equal
periodicity. Now it is time to check the inner products of sinusoids
with different periods. cos(2*pi*x) is one period, and cos(4*pi*x) is
two periods.

The product cos(2*pi*x)*cos(4*pi*x) over the interval [0,1] sums to
zero. This means that these functions are orthogonal.

Below is another one, with cos(4*pi*x)*cos(6*pi*x). Notice that the
integrated area is consistently defined over the interval [0,1]. That
is the
fundamental period, 2*pi*x, describing one full circle on the complex
plane.

4*pi*x and 6*pi*x are integer multiples, or harmonics, of this
fundamental period. They describe two and three full circles
respectively. And seem to be orthogonal to each other as well.

One more example with harmonics, three and four cycles respectively:

Oops. At first sight the product of cos(6*pi*x)*cos(10*pi*x) looks
like being evenly spread around the x-axis, like the preceding figures.
But there are no symmetric parts identifiable. From this curve,
orthogonality cannot be verified by the eye.

I made another plot where the corresponding sine parts are included.
Because sine and cosine of equal periodicity are orthogonal, I can
safely do that. Look, now there is symmetry round the x-axis again:

The product of harmonic sines and cosines has it's own periodicity. In the above figure we see that two periods of the product fit in the interval [0,1]. In earlier figures it was one period. It can also be more than two periods. The pattern is, that one or more periods of the product of two harmonics will always fit nicely in the interval [0,1]. Therefore, in the harmonic case the integral over [0,1] is well defined.

Let us now check what will happen to sinuoids with periods that can not be expressed as a
ratio of integers. Say, cos(2*pi*x) and cos(6*x). I have plotted these
two functions and their product. Here, the product curve does not show
periodicity over the interval [0,1]:

When the curves are inspected on a wider interval, they seem to show
their period. But that is illusory. The period of a sum of two cosines
can only be expressed as the product of two integers. Compare with the
beats of polyrithmic music. And try to imagine how big integer numbers
you would need to express the ratio of 2*pi and 6. It will never be
exact, but for a better approximation you will need ever bigger
numbers. That is how
the fundamental period of this sum of cosines extends into infinity.
And you would never stop summing products, although the sum would be
fluctuating around the zero line, rather than growing spectacularly....
That is
what they call an infinite sum. The inner product of such non-harmonic
sinusoids is by itself undefined.

It seems we have the following cases for the inner products of two sines or two cosines:

- they have inner product 0.5 if their period is equal

- they have inner product 0.0 if their period is a ratio of (non-equal)
integers

- they have an indefinite inner product if their combined period cannot
be expressed as a ratio of integers

These conclusions may not sound very promising. On the other hand, they
lead to a straightforward hint: fill your correlation matrix with
harmonic sines
and cosines. They are orthogonal. That is one
condition fulfilled, for a start. Oh, and do not forget about the
complex conjugates.

With these simple statements made, the step to classic Fourier
Transform is not a big step. But the story of orthogonal sinusoids is
not yet complete. I want to outline an orthogonal sinusoids set that is
a bit more obscure, because they can not be expressed as complex
numbers or complex exponentials. It is cosines only.

Below, cosine functions with period 0.5 and period 1 are plotted, and
the product of these:

These functions are not harmonics, but 'half-harmonics'. Still they
are clearly orthogonal over the interval [0,1]. Can we also go
down to a 0.25 and 0.5 period cosine pair?

No, a quarter period does not work. The product is still periodic
but not within the interval [0,1]. It would go beyond the analysis
window. So half a period is the smallest period and thus the smallest
interval. Let us check more 'half-harmonics', does the orthogonality
hold for other cases? For example, cos(pi*x)*cos(3*pi*x), which are 0.5
and 1.5 period respectively:

Seems to work but can not be verified by the eye strictly. And
cos(2*pi*x)*cos(3*pi*x), one period and
one-and-a-half period respectively:

Still works okay. Let's get bold and try 3 and 3.5 periods:

Hmmm. Works like a charm. Actually, I am doing such
half-period-product plots for the first time now. It does not appear so
obscure at all. There is something uncomfortable in it though. Dsp
without complex vectors, feels like sailing without a compass.

Cosines with half-period intervals are used in Discrete Cosine
Transform. Not surprisingly, there is also Discrete Sine Transform.
Interesting stuff. But I will stop plotting now, because this page is
growing too long.