Outlier

Outliers are data points that stand out because they are much higher or lower than the rest of the data. Outliers can disproportionately affect statistical measures such as the mean, standard deviation, and regression models, skewing results and leading to misguided conclusions. This is because statistical measures, such as the mean, standard deviation, and regression models, are sensitive to extreme values. 

For example, 
A teacher records the marks of 10 students in a math test. \(45, 48, 50, 47, 49, 46, 50, 48, 47, 95\).
Here, all the scores are around 45 – 50, but 95 is much higher than the rest.
So, 95 is an outlier because it is unusually high compared to the other data points.

Finding outliers is an integral part of data analysis because unusual values can heavily influence the results. Outliers can be identified using two main approaches: visualization techniques and statistical methods.

Using Visualization Methods
Visualization involves presenting data in graphical form, making it easier to spot patterns and detect unusual values. Two commonly used visual tools are:

1. Box Plot
A box plot displays the minimum, first quartile (Q₁), median, third quartile (Q₃), and maximum.
Any point that lies outside the “whiskers” (beyond\(Q_1 - 1.5 \times \mathrm{IQR} \quad \text{or} \quad Q_3 + 1.5 \times \mathrm{IQR} \)) is considered an outlier.
Example:
Data: 12, 14, 15, 16, 17, 18, 40
In the box plot, 40 appears far outside the upper whisker, making it an outlier.


2. Scatter Plot
A scatter plot displays individual data points on a graph. Outliers appear as points that lie far away from the general cluster.
Example:
If you plot students’ study hours vs. exam scores, and most points form a cluster, but one point. For example, 10 hours of study and only 5 marks that lies far away, that point is an outlier.

A box plot is a statistical chart that provides a visual summary of how data is distributed. Outliers in a box plot can be identified using these steps.


Step 1: Sort the data in ascending order and determine the median.

Step 2: Calculate the Interquartile Range (IQR), which represents the central 50% of the dataset.

Step 3: Determine the lower and upper bounds (also called fences) using the IQR.

Step 4: Any data point that lies below the lower bound or above the upper bound is considered an outlier.

A scatter plot is used to visualize the relationship between two continuous variables, with each point represented as a dot. In this plot, any points far from the central cluster of data are considered outliers.


Using Statistical Methods
To detect the outliers numerically, statistical techniques are used. Some commonly used methods include Z-score, DBSCAN, and the Isolation Forest algorithm.


Identifying Outliers Using the Z-Score
The Z-score method measures how many standard deviations a data point is from the mean. It is calculated using the formula:

\(Z=X-μσ\)

Where:

X = the data point

μ = mean of the dataset

σ = standard deviation of the dataset

A data point is considered an outlier if its Z-score is greater than +3 or less than –3.

The Isolation Forest algorithm is an anomaly detection technique that uses decision trees to separate data points. It works by randomly partitioning the dataset. Points that are isolated in fewer steps are considered outliers because unusual values stand out and are easier to separate from the rest.


For example,
Consider the dataset representing daily sales (in units):
50, 52, 55, 53, 58, 60, 54, 300


Most values are close to each other, between 50 and 60. But 300 is hugely different.


When the Isolation Forest algorithm creates random partitions:

• The values around 50–60 need many splits to isolate because they are very similar.
• The value 300 gets isolated very quickly because it stands out from the rest.
   

Since 300 requires fewer splits, the algorithm marks it as an outlier. 

Outliers can be identified using different methods depending on the complexity of the data, the time available, and the level of accuracy needed. Here are four commonly used methods, along with the steps:

• Sorting Method
  Step 1: Arrange the data in ascending order.
  
  Step 2: Look for values that appear unusually high or low compared to the rest.
  
  Step 3: Verify by checking whether these extreme points are significantly different from the dataset's general pattern.
  
  Step 4: Mark these extreme values as outliers.
  For example,
  Data: 10, 12, 13, 14, 15, 80
  80 is clearly separated and becomes the outlier.
  
   
• Using Visualization
  Step 1: Plot the data using a scatter plot or box plot.
  
  Step 2: Observe the data distribution.
  
  Step 3: Identify points that are distant from the central cluster (scatter plot) or outside the whiskers (box plot).
  
  Step 4: Mark these distant points as outliers.
  For example,
  In a scatter plot, if one dot lies far away from the cluster, it is an outlier.
  
   
• Statistical Outlier Detection (Z-Score Method)
  Step 1: Calculate the mean (μ) and standard deviation (σ) of the dataset.
  
  Step 2: Use the formula,
  
  \(Z= X-μσ \)
  
  Step 3: Compute the Z-score for each data point.
  
  Step 4: If a Z-score is greater than +3 or less than -3, that data point is an outlier.
  
  For example,
  If a score has Z = 3.2 → it is an outlier.
  
   
• Interquartile Range (IQR) Method
  Step 1: Arrange the data in ascending order.
  
  Step 2: Find Q₁ (first quartile) and Q₃ (third quartile).
  
  Step 3:  Calculate the \( IQR = Q₃ – Q₁\).
  
  Step 4: Compute the outlier fences:
  
  Lower Fence = \(Q₁ – 1.5 × IQR\)
  Upper Fence =\( Q₃ + 1.5 × IQR\)
  
   
• Any data point below the lower fence or above the upper wall is classified as an outlier.
  For example, 
  If the upper fence is 40, and one value is 55 → 55 is an outlier.

In this method the data is arranged in ascending order and sorting the data visually scanning the extreme values. 

Step 1: Arrange the data in ascending, that is, from small to big

Step 2: The value which is higher than the other values are considered to be the outlier 

Calculating Outlier by Statistical Outlier Detection (Z-score Method)

The z-score is calculated by using the formula,\( z =  X - μ/σ\),

Here, X is the data point

μ is the mean of the data set

σ (sigma) is the standard deviation.

If the value is greater than or less than ±3, then the value is an outlier. That is an outlier is more than 3 times a standard deviation. 

Interquartile range is the median of the half of the data set. In this method, we find the outlier by following these steps,

Step 1: Arranging the data in ascending order, that is, low from high.

Step 2: Finding the value of Q1 and Q3, Q1 is the middle value of the lower half and Q3 is the middle of the upper half.

Step 3: Calculate the value of IQR. So, \(IQR = Q3 - Q1\).

Step 4: Finding the value of lower bound and upper bound, here the lower bound = \(Q1 - 1.5 × IQR\) and the upper bound = \(Q3 + 1.5 × IQR\).

To master the topic outliers, some tips and tricks are mentioned below.
 

• Outliers can be used to spot unusual credit card transactions by comparing them against a customer’s typical spending behavior.
  
   
• In healthcare, outliers help identify abnormal vital signs in a patient’s records, which may indicate conditions that require immediate attention.
  
   
• Outliers are useful in detecting unexpected spikes in website traffic from certain locations or unusual user behavior, which could signal a bot attack or viral activity.
  
   
• In cybersecurity, outliers help flag suspicious login attempts, irregular access times, or abnormal data transfers that may point to hacking or malware.
  
   
• In manufacturing, outliers help identify defective products or irregular machine performance, ensuring consistent production quality.
  
   
• Parents can encourage children to notice patterns and identify values that “don’t belong,” helping them build intuitive understanding.
  
   
• Teachers can use box plots and scatter plots during lessons to visually demonstrate how outliers appear in a dataset.

Outlier is used in different fields such as finance, environment monitoring, cybersecurity, and so on. Let’s learn a few real-life applications of outliers. 

• To identify fraud detection in finance outlier is used, as it can identify any unusual transactions using credit cards by analyzing the spending patterns of the customer. 
  
   
• In health monitoring, an outlier is used to analyze any abnormal vital signs in the patient's medical records. It is helpful as it could indicate the health issues that need immediate attention.
  
   
• To identify the unusual spikes in website traffic from a special location or user behavior.
   

• Cybersecurity Threat Detection: Outliers help in detecting unusual login attempts, irregular access times, or abnormal data transfers in networks, which could indicate hacking or malware attacks.

• Manufacturing and Quality Control: In production, outliers can highlight defective items or abnormal machine behavior, helping industries quickly address errors and maintain product quality.