3.2 Filtering data
Gapminder and States
The Gapminder and the States datasets you have created likely include many years of data. You might want to work with multiple years, but you will likely want to work with only one year of data. The select function worked with columns in our dataset. In this section, we will look at rows. We’ll use the filter function in the dplyr package. dplyr is part of the tidyverse. There are also a few other functions that are handy here: arrange, group_by, and summarize.
This 10:00 video from the R Ladies Sydney walks you through using the filter function along with arrange.
Let’s look at an example using the States dataset and start with the basics. My States dataset is big: it has 649 variables over 20 years. It could be hard to work with. I am interested in the distribution of wealth in states. The Gini coefficient is in my dataset. I want to narrow the dataset down to only one year of data. I will need to use two steps to do this:
- Select the variables I want to use,
state,gini_coef, andincshare_top1. - Filter the data to include the year I wish to use, 2011.
We will use the pipe (%>%) to do this all at once.
## # A tibble: 50 x 3
## state gini_coef incshare_top1
## <chr> <dbl> <dbl>
## 1 Alabama 0.589 16.1
## 2 Alaska 0.567 11.6
## 3 Arizona 0.624 16.7
## 4 Arkansas 0.611 17.7
## 5 California 0.671 21.5
## 6 Colorado 0.621 18.8
## 7 Connecticut 0.675 29.9
## 8 Delaware 0.564 13.6
## 9 Florida 0.685 24.6
## 10 Georgia 0.636 16.2
## # … with 40 more rowsNote this example managed both rows—through the filter command—and columns—using the select command. So now I have a small dataset that is easy to manage and useful for learning about the specific question I want to study.
I can save this dataset if I want it for later work.
3.2.1 An application
With the data I have I can find out which states have the highest and lowest Gini coefficients in 2013. To do this I need to add one step to my function, using arrange discussed in the video:
my_usstate_data %>%
filter(year == 2011 & st != "DC") %>%
select(state, gini_coef, incshare_top1) %>%
arrange(desc(gini_coef))## # A tibble: 50 x 3
## state gini_coef incshare_top1
## <chr> <dbl> <dbl>
## 1 New York 0.691 30.1
## 2 Florida 0.685 24.6
## 3 Nevada 0.681 23.1
## 4 Connecticut 0.675 29.9
## 5 California 0.671 21.5
## 6 Texas 0.637 21.1
## 7 Massachusetts 0.637 24.3
## 8 Georgia 0.636 16.2
## 9 Illinois 0.629 19.8
## 10 Idaho 0.629 14.1
## # … with 40 more rowsmy_usstate_data %>%
filter(year == 2011 & st != "DC") %>%
select(state, gini_coef, incshare_top1) %>%
arrange(gini_coef)## # A tibble: 50 x 3
## state gini_coef incshare_top1
## <chr> <dbl> <dbl>
## 1 West Virginia 0.551 15.3
## 2 Iowa 0.555 11.8
## 3 Maine 0.559 12.8
## 4 Delaware 0.564 13.6
## 5 Hawaii 0.566 10.6
## 6 Alaska 0.567 11.6
## 7 Ohio 0.568 15.1
## 8 Kentucky 0.574 14.2
## 9 Indiana 0.579 14.2
## 10 Wisconsin 0.579 15.5
## # … with 40 more rowsSo New York has the highest Gini coefficient and, West Virginia has the lowest. We can also see that the top 1% of New Yorkers have 31.8% of the state’s wealth.
Let’s compare the Gini coefficient in New York to countries and the United States as a whole using Gapminder data.
my_gapminder_data %>%
filter(year == "2013") %>%
select(country, giniCoefficent) %>%
arrange(desc(giniCoefficent))## # A tibble: 163 x 2
## country giniCoefficent
## <chr> <dbl>
## 1 South Africa 63.2
## 2 Botswana 60.5
## 3 Namibia 60
## 4 Zambia 56.5
## 5 Central African Republic 56.2
## 6 Lesotho 54.2
## 7 Belize 53.3
## 8 Honduras 53
## 9 Colombia 52.7
## 10 Brazil 52.2
## # … with 153 more rowsmy_gapminder_data %>%
filter(year == "2013" & country == "United States") %>%
select(country, giniCoefficent) %>%
arrange(desc(giniCoefficent))## # A tibble: 1 x 2
## country giniCoefficent
## <chr> <dbl>
## 1 United States 41New York’s Gini coefficient of 69.9 is higher than any country and is much higher than the United States as a whole (41).
