Central Limit Theorem

The central limit theorem states that if we consider numerous random samples from a population, the sample mean distribution will form a normal distribution, which is a bell curve. The distribution of the original population can be skewed, Poisson, or binomial, but the mean distribution of the sample will be normal when the sample size is greater than or equal to 30. So, if the sample size is larger, it allows for better estimation of population characteristics.

Central Limit Theorem Diagram

The Central Limit Theorem diagram clarifies a counterintuitive concept: how random chaos becomes predictable order. It visually demonstrates that no matter how weird or skewed the original "parent" population looks, the distribution of its sample averages will always shape-shift into a perfect Bell Curve.
 

So far, we have discussed what the central limit theorem states. Let us now learn about the key components of the theorem. The key concepts are: 
 

• Sampling is successive: It means that some samples may repeat. For instance, in opinion polls, the population selected can be a part of different studies. 
   
• Sampling is random: The sample here must be selected randomly, as it gives equal chance to all the individuals in the population. For example, in a political poll, if we select the sample based on a specific group, the result can be biased.
   
• Samples should be independent: Each sample should be selected individually; the selection of one should not influence the other. The sample should be independent to follow a true, normal pattern.
   
• Large sample size: If the sample is large, the sampling distribution will be normal. According to the central limit theorem, an ideal sample size should be 30 or more. 

Here are the four key assumptions of the Central Limit Theorem (CLT) in point form:

• Independence Assumption: The sampled observations must be independent of each other. One data point should not influence the next.
   
• Randomization Condition: The data must be collected using a random sampling method or a randomized experiment to avoid bias.
   
• Sample Size Condition: The sample size n must be “large enough.” The general rule of thumb is \(n \ge 30\), though skewed populations may require larger samples.
   
• Finite Variance: The population from which the sample is drawn must have a finite mean (\(\mu\)) and a finite variance (\(\sigma^2\)).
   

Now let’s look at the formula for the central limit theorem. Let X be a random variable with a known or unknown probability distribution. The standard deviation is σ, and the mean of X is μ. According to the central limit theorem, if the large number sample are drawn of size n, then the new random variable is x̄, then,

\(\bar{x} \sim N\!\left(\mu, \frac{\sigma}{\sqrt{n}}\right)\), where σ / √n is the standard error. The z score of the random variable x̄ is \( z = \frac{\bar{x} - \mu}{\sigma / \sqrt{n}} \)

Below is the step-by-step proof of the Central Limit Theorem.

1. Setup and Assumptions

Let \(X_1, X_2, ..., X_n\) be a sequence of independent and identically distributed (i.i.d) random variables with:
 

• Mean: \(E[X_i] = \mu\)
• Variance: \(Var(X_i) = \sigma^2\) (where \(\sigma^2 < \infty\))
• Moment Generating Function: \(M_{X}(t)\) exists in a neighborhood of 0.
   

We want to prove that the standardized sum converges to a Standard Normal Distribution \((Z \sim N(0,1))\) as \(n \to \infty.\)

2. Standardization

First, we shift and scale the variables to make the math cleaner. Let \(Y_i \) be the standardized version of \(X_i\):

\(Y_i = \frac{X_i - \mu}{\sigma}\)
 

Consequently, the properties of \(Y_i\) are:

\(E[Y_i] = 0\)

\(Var(Y_i) = E[Y_i^2] - (E[Y_i])^2 = 1\)

We define the statistic \(Z_n\) (the standardized sample mean) as:

\(Z_n = \frac{\bar{X} - \mu}{\sigma/\sqrt{n}} = \frac{1}{\sqrt{n}} \sum_{i=1}^{n} Y_i\)
 

3. Express the MGF of \(Z_n\)

We need to find the MGF of \(Z_n\), denoted as \(M_{Z_n}(t)\).

\(M_{Z_n}(t) = E\left[ e^{t Z_n} \right] = E\left[ e^{t \frac{1}{\sqrt{n}} \sum Y_i} \right]\)
 

Using the property of exponentials \((e^{a+b} = e^a e^b)\) and the independence of \(Y_i\):

\(M_{Z_n}(t) = E\left[ \prod_{i=1}^{n} e^{\frac{t}{\sqrt{n}} Y_i} \right] = \prod_{i=1}^{n}E\left[ e^{\frac{t}{\sqrt{n}} Y_i} \right]\)

Since all \(Y_i\) are identical, we can write this in terms of the MGF of a single Y:

\(M_{Z_n}(t) = \left( M_Y\left( \frac{t}{\sqrt{n}} \right) \right)^n\)

4. Taylor Series Expansion

We expand \(M_Y(s)\) using a Taylor series around 0.

\(M_Y(s) = M_Y(0) + M_Y'(0)s + \frac{M_Y''(0)s^2}{2} + o(s^2)\)
 

Recall the moments of Y:
\(M_Y(0) = 1\)
\(M_Y'(0) = E[Y] = 0\)
\(M_Y''(0) = E[Y^2] = 1\)

Now, substitute \(s = \frac{t}{\sqrt{n}}\) into the expansion:

\(M_Y\left( \frac{t}{\sqrt{n}} \right) = 1 + 0 \cdot \frac{t}{\sqrt{n}} + \frac{1}{2} \left( \frac{t}{\sqrt{n}} \right)^2 + o\left( \frac{t^2}{n} \right)\)

\(M_Y\left( \frac{t}{\sqrt{n}} \right) = 1 + \frac{t^2}{2n} + o\left( \frac{1}{n} \right)\)

5. Taking the Limit

Now substitute this back into our equation for \(M_{Z_n}(t)\) and take the limit as \(n \to \infty\):

\(\lim_{n \to \infty} M_{Z_n}(t) = \lim_{n \to \infty} \left( 1 + \frac{t^2/2}{n} \right)^n\)

Recall the standard calculus limit definition of \(e^x\): \(\lim_{n \to \infty} (1 + \frac{x}{n})^n = e^x\).

Here, \(x = t^2/2\).
 

Therefore:

\(\lim_{n \to \infty} M_{Z_n}(t) = e^{t^2/2}\)

6. Conclusion

The function \(e^{t^2/2}\) is the Moment Generating Function of the Standard Normal Distribution.
 

By the Uniqueness Theorem, since the MGF of \(Z_n\) converges to the MGF of the Standard Normal distribution, the distribution of \(Z_n\) must converge to the Standard Normal distribution. Q.E.D.

Here are the key properties of the Central Limit Theorem:

• Normality of Averages: The most famous property. As you take more samples (n), the distribution of the sample means will form a Normal Distribution (Bell Curve), even if the original data is weird, skewed, or flat.
   
• Equality of Means: The average of all your sample means will equal the true Population Mean (\(\mu\)). This makes the sample mean an "unbiased estimator."
   
• Reduction of Variance: The spread of your sample means is always smaller than the spread of the original population. It is calculated as \(\frac{\sigma}{\sqrt{n}}\).
   
• The “Square Root” Rule: To cut your error in half, you must quadruple your sample size (because of the \(\sqrt{n}\) in the formula).
   
• Independence: The theorem only holds if each sample is independent of the others (one sample doesn't change the probability of the next).
   
• Sample Size Threshold: For most non-normal populations, a sample size of \(n \ge 30\) is considered the “magic number” where the CLT properties fully kick in.
   
• Finite Variance: The population you are sampling from must-have a defined, finite variance. (It doesn't work for theoretical distributions with infinite variance, like the Cauchy distribution).
   

As we know, the central limit theorem is a fundamental concept in statistics and probability; it helps us understand how the population estimates the behavior under repeated sampling. Now we will discuss how it works and the formulas. 

Step 1: Identify the Setup

Write down the Sample Size (n), Population Mean (\(\mu\)), and Standard Deviation (\(\sigma\)). Identify the inequality in the question:

• Greater than (>): “exceeds”, “at least”, “more than”
• Less than (<): “at most”, “below”, “under”
• Between: “from X to Y”, “between”

Step 2: Draw the Graph

Sketch a Normal Distribution curve with the Mean (\(\mu\)) in the center.

• Mark your sample mean value (\(\bar{x}\)) on the axis.
• Shade the area corresponding to the inequality (shade right for >, left for <, or the middle for “between”).

Step 3: Calculate the Z-Score

Apply the CLT formula to convert your value into a Z-score.

\(Z = \frac{\bar{x} - \mu}{\sigma / \sqrt{n}}\)

(Note: For “Between” problems, you must calculate two Z-scores: one for the lower limit and one for the upper limit.)

Step 4: Look up the Probability

Locate the calculated Z-score in the Z-table to find the corresponding Probability Area (Decimal Value). (Note: Most standard tables give the area to the left of the Z-score).

Step 5: Adjust for Direction (Crucial Step)

Use the decimal value from Step 4 based on your problem type:

• For Less Than (<): The answer is the value from the table.
• For Greater Than (>): The answer is \(1 - \text{Table Value}\).
• For Between: Find the table values for both Z-scores. The answer is \((\text{Larger Area}) - (\text{Smaller Area})\).

Step 6: Sanity Check

Look at your graph from Step 2.

• Does your shaded area look bigger than 50%? Make sure your answer is > 0.5.
• Does it look small? Make sure your answer is < 0.5.

Step 7: Final Conversion

Convert the final decimal value into a percentage if required (multiply by 100).

The Central Limit Theorem is often the most confusing concept in introductory statistics because it is abstract. It is counterintuitive—it claims that even if your data is messy and random, the averages of that data will naturally form a perfect, orderly pattern. Here are some tips and tricks to help you with the concept:

1. The Phone Number Trick: Have students average the last 3 digits of their phone number. Plotting the class averages will always create a Bell Curve, even though the digits themselves are random.
    
2. The Pizza Analogy: Explain that one pizza delivery might be super late (an outlier), but the average delivery time of 50 pizzas will almost always be on schedule. Large samples dilute extreme events.
    
3. The Galton Board Visual: Show them a Galton Board (balls dropping through pegs). It proves physically that random chaos naturally stacks up into a perfect order (the Bell Curve) in the middle.
    
4. Wisdom of Crowds: Ask a group to guess the number of jellybeans in a jar. Individual guesses will be wild, but the average of all guesses is usually scarily accurate.
    
5. The “Rule of 30”: Use dice. Rolling one die is random (flat graph). Summing 30 dice creates a perfect bell shape. It teaches that you don't need infinite data—30 is often enough.
    
6. The “Scoop” Concept: Tell them the CLT isn't about the whole bucket of ice cream (Population); it's about the average of every spoon you scoop out (Sample Mean).
    
7. Visual Check: Teach them that a “Skinny” Bell Curve means a huge sample size (high confidence), and a “Fat” Bell Curve means a small sample size (low confidence).

The central limit theorem is used in the real world to analyze the population characteristics. Let’s discuss the real-world applications of the central limit theorem. 

 

• In economics and data science, we use central limit theorem to understand the statistical model and to draw conclusion.
  
  
   
• To understand the public opinion, we use the central limit theorem to analyze and understand the polls and surveys.