Comparing Distributions

00:01 Welcome to Lecture 4, where we're gonna discuss comparing the distributions of two different quantitative variables.

00:07 The big picture here, is how do we describe these distributions? When we talk about what the distribution looks like, we need to use two key components described in the previous chapter.

00:18 Graphical displays, to get an idea of the shape.

00:21 Summary statistics, to get an idea of where the distribution centered and how spread out it is.

00:27 For example, if we were measuring average wind speed for each day through out the year we would describe the distribution of wind speeds by saying, "Here's a histogram, a 5-number summary, and a boxplot." Suppose wind speeds are unimodal and skewed right, we would report the median and the interquartile range.

00:46 We can compare the distributions of multiple groups by looking at histograms of each group.

00:51 And what can this tell us? Well, from this, we can learn how the shapes of the distributions differ.

00:57 We can learn whether or not, one appears to be more spread out that the other.

01:01 And we can also learn whether or not, one distribution appears to be centered in a different place than the other.

01:08 For instance we might ask, "Is it windier in the summer, than it is in the winter?" And this question can be approached by looking at histograms of the wind speeds in the summer, and histograms of the wind speed in the winter.

01:20 All right, so let's look at an example dealing building a nest egg.

01:25 And we're gonna look at two different regions of the country.

01:28 The nest egg index is a measure of savings for retirement for each of the 50 states.

01:35 And the states were divided into two regions, the South and the West, and the North and the Midwest.

01:40 And the question is, "How do the Nest Egg indices compare between the two regions?" So we're gonna look at histograms for the nest egg indices between the two groups, and we'll compare the shape, center, and spread for each of the groups.

01:54 So here's a histogram for the nest egg index for the South and West region, and we see that it's fairly symmetric, and unimodal, and centered right around -- it appears to be around 97.

02:06 Here's a histogram of the nest egg index for the North and the Midwest region, and we see that we have a bi-modal distribution that has a strong left skew and appears to be centered around a hundred.

02:17 Here's the comparison of the two distributions.

02:21 For the South and West, like we saw, it's slightly skewed to the right, and unimodal.

02:25 But for the Northeast and Midwest, we have a strong left skew and a bi-modal distribution.

02:32 So that's what the histogram tells us.

02:34 We saw that the South and West distribution appears to be centered around 95, and that the Northeast and Midwest distribution appears to be centered around 100 or 105.

02:45 We also saw that there appears to be more spread in the Northeast and Midwest region, than there was in the South and the West region.

02:53 So let's compare these two distributions using numerical summaries.

02:57 So we can assess what we think about the center, and the spread of these distributions by using the summary statistics.

03:03 So our measures of center, and spread to compare them.

03:06 Both distributions are skewed, so it's best to report the median and interquartile range for each.

03:12 For the South and the West region, we found that the median is 97.

03:16 The median for the Northeast and the Midwest region is 103.

03:23 For the South and the West region, the interquartile range is 6.5, whereas for the Northeast and Midwest region, the interquartile range is 8.5.

03:34 So it turns out that our interpretations of the histograms is pretty accurate because the Northeast and Midwest region is more spread out, than the South and the West distribution.

03:44 And the centers of the two distributions are about where we thought they were based on the histograms.

03:50 We can also compare these groups using boxplots.

03:54 We put these boxplots side-by-side, and sometimes this gives a better picture of the differences between the two distributions.

04:00 It can give it some sense of how the shapes differ, in terms of the symmetry and the skewness.

04:06 But the problem with boxplots is if they hide modes, so we won't be able to see that the South and West distribution is bi-modal from the boxplot.

04:14 It is extremely useful as a way to compare center, and spread between the two distributions.

04:21 So we place them side-by-side in the same plot.

04:24 So here's the example of the side-by-side boxplots.

04:27 We have, for the South and West region, that the center is, again, around 97, and that there's less spread in the South and the West region than there is in the Northeast and the Midwest region.

04:41 And these things can be easily ascertained from looking at a boxplot.

04:45 The boxplot show two of the key features that we got from the histograms.

04:51 Its shows the differences in the center, and it shows the differences in the spread.

04:56 But we lose some of the information that we got from the histogram.

05:00 We can't see the two peaks in the Northeast and Midwest region.

05:04 But we can still see the left skew.

05:07 So the big picture here is that boxplots are great for determining whether there are differences between the center and spread, but not so reliable in terms of determining shapes.

05:17 So when you're comparing distributions, it's best to look at both side-by-side histograms, and side-by-side boxplots.

05:24 What about outliers? What are they, and how do they affect our description of the data? When we talk about outliers, what we're talking about are observations that are far away from the other values in the data set.

05:36 Basically, extreme values.

05:37 And there are several ways that these can occur, they can occur due to a data entry error, or other misreports of the data values.

05:44 But they might also be important, they might highlight exceptional cases.

05:48 Like in the South and Midwest region, if we're looking at wind speeds at this time.

05:54 Maybe on one day, there's a tornado and the wind speed is very high.

05:59 Well, that's important, that's not data entry error.

06:02 That's important information, so that would highlight in the exceptional case.

06:05 Boxplots help us see outliers, those are the dots that we put outside the fences if we have extreme observations.

06:12 Often, histograms can show outliers, and they're usually represented by bars that are kinda next to gaps in the histogram.

06:20 So here's an example of the South and West nest egg data with an outlier added.

06:27 We added an extra observation with the nest egg index of 150.

06:32 And so here's the original boxplot of the South and West region, and then there's that dot way up at the 150 to symbolize the outlier.

06:40 Now, if you look at the outlier with the histogram, we see that there's that one observation way out to the right side, far away from everything else in our data set.

06:50 So from the histogram, we can clearly see that this is an outlier.

06:54 Sometimes, we wanna make our data more symmetric, so that we can deal with the mean and the standard deviation as center, and spread measures as opposed to the median and the IQR.

07:04 And one way to do that is to transform our data set.

07:07 If the data are skewed, it can be difficult to summarize our distribution using center and spread.

07:14 And it can also be difficult to determine whether the most extreme values are outliers, or if they're just part of the stretched out tail.

07:22 So transforming data is one way to get around this problem.

07:26 So for our data X1 through Xn, and their distribution is strongly skewed, one way to make the distribution a little bit more symmetric is to use a logarithmic transformation, so that what we're working with as our data values are actually natural logs of the data values that we actually observed.

07:45 So again, here's the histogram of the South and West data as we had them before.

07:50 And now we're gonna look at the histogram of the natural log of the South and West nest egg indices, and we see that we have a little bit more symmetry here than what we did before.

08:02 What happened to the histograms? We do have more symmetry after we do the transformation, but transforming the data will not always fix the lack of symmetry.

08:14 If the data are extremely skewed, then the log transformation is unlikely to be very helpful.

08:21 But if you have slight skew then the log transformation can be useful in fixing that problem.

08:26 We have another common transformation which is is the square root transformation.

08:30 And so instead of working with our original data set, what we're gonna work with is the square roots of our original observations.

08:38 What about equalizing the spread between two groups? Maybe we don't want data that are -- that have one group more spread out than the other.

08:49 So there's a whole bunch of statistical procedures for comparing two groups that rely on the assumptions that the variance for the first group is the same as the variance of the second group.

08:59 And this is one reason that we might wanna equalize the spread between the two groups.

09:03 The logarithmic transformation can often be used, also, to equalize the spread between the two groups in order to use the procedures that rely on that assumption.

09:13 But other common transformations that we use for this, is the square root transformation, and the inverse sine transformation.

09:21 But the inverse sine transformation is far less common that the square root, because they both kinda do the same thing in terms of equalizing variance.

09:28 Okay, so here's a boxplot of log transformed data for the nest egg indices.

09:33 We have the South and West region on the left-hand side.

09:36 The Northeast and the Midwest region on the right-hand side.

09:39 And what do we see? We see that the spread in the boxes is a lot more similar than what it was before.

09:46 The Northeast and Midwest region is still just a little bit more spread out than as the South and West, but that difference is not as great as it was before.

09:54 All right, so when we compare distributions, we have several issues that we can run into.

10:00 One of the biggest problems that we run into is using inconsistent scales.

10:05 We always wanna compare our distributions for variables on the same scale.

10:09 In other words, don't transform one group, and not the other.

10:13 We wanna make sure that our plots are clearly labeled, because a plot is of little use if it's unclear what information it's giving you.

10:22 If we don't know what information this plot is supposed to convey, then we might as well not have a plot at all.

10:28 So labeling the plot clearly fixes that problem.

10:32 This is very important, beware of outliers.

10:36 If the data have outliers, and you can correct them, then do it.

10:40 If they're clearly errors, then take them away.

10:43 But otherwise, summarize the data twice, do it once with the outlier in it, and once with the outliers removed.

10:50 And that's the end of Lecture 4, Comparing Distributions. And we'll see you back here for Lecture 5.