Tests of Significance
Summary of the Video
When a sample of students take a coaching course to prepare for the Scholastic Aptitude Test, their scores improve more than the scores for a control group of students who aren't coached. Is this good evidence that coaching improves SAT scores? There's chance involved in assigning students to the coaching group or the control group. Maybe the coaching group got students whose next try at the SAT was going to be a big improvement—just by chance.
Statistical inference draws conclusions about a population on the basis of sample data. One important kind of inference assesses the strength of the evidence given by a sample for some conclusion about the population. Do the results of a sample convince us that coaching helps SAT scores on the average for all students? Significance tests answer questions like this by telling us if the observed results could plausibly occur just by chance.
We look at taste testing of NutraSweet, an artificial sweetener used in diet colas. The video shows the careful assessment of sweetness by trained tasters, first when the cola is fresh and then after a month of storage at high temperature that imitates the effect of four months on store shelves. Our data are the differences in sweetness scores (before minus after storage) for 10 tasters:
2.0 0.4 0.7 2.0 –0.4 2.2 –1.3 1.2 1.1 2.3.
The average sweetness loss is given by the sample mean,
That's not a really big loss. If we tried again we would get a somewhat different result. Maybe it's just chance that produced this result. A test of significance asks:
Does the sample result = 1.02 reflect a real change in the cola?
Could we easily get the outcome = 1.02 just by chance?