Follow us on twitter @simplystats
Follow us on twitter @simplystats
This is awesome. There are a few places with some strong language, but overall I think the message is pretty powerful. Via Tariq K. I agree with Tariq, one of the gems is:
If you want to measure something, then don’t measure other sh**.
Via AL Daily, this dude collects data and analyzes it to put war criminals away. The idea of using statistics to quantify mass testimony is interesting.
With statistical methods and the right kind of data, he can make what we know tell us what we don’t know. He has shown human rights groups, truth commissions, and international courts how to take a collection of thousands of testimonies and extract from them the magnitude and pattern of violence — to lift the fog of war.
So how does he do it? With an idea from statistical ecology. This is a bit of a long quote but describes the key bit.
Working on the Guatemalan data, Ball found the answer. He called Fritz Scheuren, a statistician with a long history of involvement in human rights projects. Scheuren reminded Ball that a solution to exactly this problem had been invented in the 19th century to count wildlife. “If you want to find out how many fish are in the pond, you can drain the pond and count them,” Scheuren explained, “but they’ll all be dead. Or you can fish, tag the fish you catch, and throw them back. Then you go another day and fish again. You count how many fish you caught the first day, and the second day, and the number of overlaps.”
The number of overlaps is key. It tells you how representative a sample is. From the overlap, you can calculate how many fish are in the whole pond. (The actual formula is this: Multiply the number of fish caught the first day by the number caught the second day. Divide the total by the overlap. That’s roughly how many fish are really in the pond.) It gets more accurate if you can fish not just twice, but many more times — then you can measure the overlap between every pair of days.
Guatemala had three different collections of human rights testimonies about what had happened during the country’s long, bloody civil war: from the U.N. truth commission, the Catholic Church’s truth commission, and the International Center for Research on Human Rights, an organization that worked with Guatemala’s human rights groups. Working for the official truth commission, Ball used the count-the-fish method, called multiple systems estimation (MSE), to compare all three databases. He found that over the time covered by the commission’s mandate, from 1978 to 1996, 132,000 people were killed (not counting those disappeared), and that government forces committed 95.4 percent of the killings. He was also able to calculate killings by the ethnicity of the victim. Between 1981 and 1983, 8 percent of the nonindigenous population of the Ixil region was assassinated; in the Rabinal region, the figure was around 2 percent. In both those regions, though, more than 40 percent of the Mayan population was assassinated.
Cool right? The article is worth a read. If you are inspired, check out Data Without Borders.
Here are a few ideas that might make for interesting student projects at all levels (from high-school to graduate school). I’d welcome ideas/suggestions/additions to the list as well. All of these ideas depend on free or scraped data, which means that anyone can work on them. I’ve given a ballpark difficulty for each project to give people some idea.
Happy data crunching!
Data Collection/Synthesis
Data Analyses
Data visualization