Imagine you're watching A Christmas Story, and Ralphie just opened up his brand-new Red Ryder BB Gun on Christmas Day. Red Ryder claims that their guns shoot at an average of 350 feet per second (fps)... talk about some velocity!
However, in this edition of A Christmas Story, Ralphie is a young prodigy in statistics. Being the intellect that he is, he understands that every shot from a Red Ryder BB Gun is not going to be exactly 350 fps... there's obviously going to be some wiggle-room there. In addition, the true average velocity of all Red Ryder BB Guns produced might not be exactly 350 fps... it might be a couple decimal points different.
To help the Red Ryder community, he wants to determine an interval of fps values that one can be a certain degree confident a Red Ryder BB Gun will consistently shoot.
Herein lies the purpose of a confidence interval...
A confidence interval enables one to obtain a range of values in which the true population parameter lies, with a defined confidence level.
Without digging into the math yet, let's say Ralphie collected a random sample of 35 Red Ryder BB Guns and found an average fps of 355.2 with a standard deviation of 5.4 fps. Based on these numbers, we can compute the following 95% confidence interval (CI):
CI = (353.345, 357.055)
This means we're 95% confident that the true mean fps lies within this interval for all Red Ryder BB Guns of the same model. We are not saying that there's a 95% probability... rather that we are 95% confident.
The crucial difference between "probability" and "confidence"
What's the difference between "probability" and "confidence" here?
Either μ lies in the interval or it doesn't. There is no "probability" about it. The process by which the interval is derived leads to coverage in 95% of cases over the long run.
In other words, the true population mean already exists. There's not a "probability" of it being a certain value.
However, we often can't know for certain what that value is. It's often impossible to gather all the data from a given population to get that true population mean. That's why we must gauge our confidence that it lies between a range of values!
The population mean already exists. There's no probability of it being in a certain value. That's why we gauge our confidence of it lying between a range of values, since we often can't know for certain where the actual value lies.
How can we be that confident?
In the situation above, there are millions of different combinations of 35 Red Ryder BB guns that we could've sampled. Due to limitations on time and resources, we were limited to only taking one sample.
Imagine, however, we took 20 different samples (of 35 Red Ryder BB guns each) instead of 1. Each of those samples would produce different sample means and different confidence intervals, because each sample would be composed of different BB guns.
And... a certain percentage of them would contain the true population mean!
But, what percentage would that be?
In our case, it'd be 95%, since that was our confidence level. Notice how 95% (so 19) of the intervals we found from our 20 different samples contain the true population mean...
...and 5% (so 1) don't contain it!
Your confidence interval percentage / confidence level determines what percentage of random samples of the same size would contain the true population parameter.
If you'd like deeper reading on this topic of understanding the confidence level, check out this article from Statistics by Jim!
What impacts the range that our confidence interval covers?
There's two factors that impact the size of our confidence interval: sample size and confidence level.
Simply put, the more data points you have, the more confident you will be in your findings!
For example, imagine you're at a bar and get asked to play a game of darts. The issue is... this is will be your very first time throwing a dart.
What if before you threw it, you were asked to give the general area in which you were 95% confident the dart would land? For the sake of the example, let's say you're 95% confident you'll hit anywhere on the dart board. (Not 100% confident though. Remember, it's your first time throwing a dart!)
But, as you throw more and more darts (increasing your sample size), you become more and more skilled, and the general area where you are 95% confident the dart will land becomes smaller, because you're getting more experience throwing and becoming more accurate!
The bigger your sample size (while holding confidence level constant), the narrower your confidence interval will be.
Imagine instead of being the one throwing the dart, you're a bystander watching from the bar. You've been watching the game for a little bit, and most of the darts are landing on the board, but with very little accuracy.
After a minute, the bartender leans over and asks how confident you are that this next dart will land anywhere on the board. Judging by how the game has gone thus far, you respond that you're 90% confident. The entire board is a wide range for the dart to land, so you're pretty confident.
What if instead, the bartender asked how confident you were that the next dart would be a bullseye? Judging by the lack of accuracy displayed, you'd probably respond that you're 1% confident, since now the dart has a much narrower range to land.
The higher your confidence level (while holding sample size constant), the wider your confidence interval will be.