Covariance Matrix

A covariance matrix is used in statistics to understand the types of relationships between different variables. The matrix gives us variance and covariance. Variance refers to the measure at which a variable expands from its mean, and covariance tells us how two variables change with respect to each other.
Covariance can be positive, negative, or zero. A positive value suggests both variables increase together. A negative value means when one variable increases, the other decreases. A zero covariance suggests that the variables are not related.
 

Follow the given steps to calculate a covariance matrix when a dataset is provided:

Step 1: Organize the dataset forming an n × m matrix with each row representing an observation or data point and each column representing a variable.   
For example:
  

Step 2: Find the mean of each column.

Step 3: Subtract the mean of each column from every entry in that column. This results in a mean-centered matrix. xcentered​ = x − xˉ

Step 4: Apply the covariance matrix formula.
Covariance matrix =1n-1xcenteredTxcentered
Here,
xT is the transpose of the centered matrix 
 1n-1 is used to sample the covariance
A centered matrix is a matrix from which the mean of each variable is subtracted from its values, resulting in each column having a mean of zero.
 

The properties of the covariance matrix are listed below:

1. The covariance matrix is always a square matrix of order n × n order. 
2. Covariance matrices are always symmetric and follow Cov(xi, xj) = Cov(xj, xi)
3. Each diagonal element in a covariance matrix is a variance, ii=Var(xi)
4. All off-diagonal elements in a covariance matrix are covariances, ij=Cov(xi,xj)
5. For any non-zero vector a, aTa  0.
6. The eigenvalues are real and not complex because the covariance matrix is symmetric. 
7. Symmetric matrices have a full set of orthogonal eigenvectors.
8. The linear transformation rule in covariance matrix suggests that if Y = ax + b, then Cov(y) = a  Cov(x)  aT.
9. If a variable has no variations, i.e., it is a constant, then its variance is zero. Its covariance with other variables is also zero resulting in a zero covariance matrix.
10.  If the variables in a covariance matrix are independent, then the matrix becomes diagonal and covariance between them is zero.
     

For a random vector with n variables X = [x1, x2, . . ., xn]. The covariance matrix is an n × n square matrix:
      
Where, 
Sample variance: var(x) = 1n(xi-x ⎺)2n - 1
Sample covariance: cov(x, y) = 1n(xi-x ⎺)2(yi-y ⎺)n-1
Population variance: var(x) = 1n(xi-)2n
Population covariance: cov(x, y) = 1n(xi-x)(yi-y)n
Here, 
 represents the mean of the population.
x ⎺ is the mean of the sample data.
n is the total number of observations in the dataset.
xi refers to individual data points in the dataset x

2 ⨯ 2 Covariance Matrix
For two random variables x and y, a 2 × 2 matrix is expressed as 

3 ⨯ 3 Covariance Matrix
For three random variables x, y, and z a 3 × 3 covariance matrix is represented as:
 
 

Covariance matrices help in understanding the relationship between variables and are used across many fields, in real-life situations, including the following:

Risk analysis of stocks in finance 
Investors use covariance matrices to understand how different stocks will rise or fall in the market. This is useful for diversifying investment and spreading it across different assets.

Principal component analysis in machine learning 
PCA requires covariance to identify the main directions in which the data varies. This helps simplify the data while focusing only on those directions, like compressing an image without affecting the quality.

Noise reduction in image processing
Nearing pixels in an image are often similar. Covariance matrices help isolate and reduce random noise that may affect image quality. They help retain image quality, which is useful in MRI and CT scan processing.

Object tracking in engineering and robotics
In robotics and engineering, covariance helps in tracking movements of an object. This property can be seen in Kalman filter used to predict where a moving object is going even when an image is unclear.

Climate pattern detection in environmental sciences
Covariance matrices help study how temperature, rainfall, or pressure readings across different regions relate to each other.This is useful for identifying patterns like El Niño or predicting future climate changes.