Adventures in research framing and promotion

The press release headline for a new paper reads “Study: Low status men more likely to bully women online”, and the original article (ungated) is titled “Insights into Sexism: Male Status and Performance Moderates Female-Directed Hostile and Amicable Behaviour”.

What is the source of these deep insights into sexism? As SirT6 put it on r/science,

“Low status” in the context of this research means guys who are bad at Halo 3.

The headlines, and the introductory portions of both the popular and academic versions of the article, state findings that are much more forceful and general than the actual article. (Incidentally the press release is also by the original authors, so they are responsible for what it says.) The core finding of the study is that the number of positive comments made by male players towards female players goes up as the male players skill rises. They find nothing for negative comments. It’s pretty hard to sell.

I find the paper itself pretty interesting, and I do think the results fit the authors’ hypotheses. I do take issue with their inclusion of players’ in-game kills and deaths as predictors in their regression: those are outcome variables, and putting them in your regression on the right-hand side will lead to biased coefficient estimates. I also think the graphs overstate their findings and should show the binned values of the actual datapoints, and the confidence intervals for their fitted curves. But it’s a nice contribution in a clever setting and it has value.

What I see as a much bigger problem with the paper is how the research is being pitched, and it’s really not the authors’ fault at all. A lot of media attention goes toward the systematic biases in scientific research due to publication bias and selective data analysis choices, but there is a huge elephant in the room: framing and salesmanship. The success of a paper often depends more on how the authors frame it (relative to the audience of the journal, their specific referees, and the wider public) than on the fundamental scientific value of the work. In part, this is another way of saying that successful scientists need to be good writers.

But it also means that there is an incentive to oversell one’s work. I see this quite a bit in economics, where papers that are fairly limited in their implications are spun in a way that suggests they are incredibly general. All the incentives point in this direction: it’s hard to make it into a top general-interest journal without claiming to say something that is generally interesting.

The spin here is particularly strong: the paper that shows that having a female teammate causes male Halo 3 players to make more positive comments if they are high-skilled and fewer if they are low skilled, and got sold as saying that males online in general engage in more bullying if they are low-status. It’s often more subtle, leaving even relatively knowledgeable readers clueless about the limitations of a work, and leading to conclusions about a topic that are unsupported by actual evidence.

Posted in Uncategorized | Leave a comment

Bad economics and rhino conservation

A firm called Pembient has a plan to combat rhino poaching by making synthetic rhino horn using keratin and rhino DNA. They can undercut prices for the real stuff by a long ways, and nearly half of a sample of people in Vietnam said they’d be willing to switch to their fake version. And they can do this without killing any rhinos. Sounds like something we should all support, right? Not according to the International Rhino Foundation. It’s executive director, Susie Ellis, makes the following pair of specious claims:

Selling synthetic horn does not reduce the demand for rhino horn [and] could lead to more poaching because it increases the demand for “the real thing.”

Actually, yes it would reduce demand for real rhino horn. These two products are substitutes, and nearly perfect ones at that. This is quite literally Econ 101.

The second part of the statement is simultaneously less wrong and more ill-founded. It’s not wrong, in the sense that I am sure no one has systematically studied whether access to synthetic rhino horn (a product that does not yet exist) helps give people stronger preferences for the “real thing” (a product that is illegal and must be damnably hard to do research on). But this objection is based on nothing more than idle speculation.

Ellis goes on:

And, importantly, questions arise as to how law enforcement authorities will be able to detect the difference between synthetic and real horn, especially if they are sold as powder or in manufactured products.

What she is proposing is the ideal outcome for the synthetic-horn intervention. They are going to sell the fake stuff at one-eighth the price of real horns. If the difference is undetectable, that would eliminate the strong incentives poachers have to kill rhinos.

I am not claiming that none of these things will happen a little bit. Rather, my point is that for Pembient’s plan to be harmful on net, all of these other potential effects that Ellis came up with – the idea that synthetic rhino horn is somehow not a substitute for real horn, that it is a “gateway drug” to the real thing, and that it would provide a way for real horn purveyors to hide their ill-gotten goods – would have to overwhelm a massive drop in price for rhino horn, which would drastically reduce supply of the real, more expensive thing.

The alternative is the status quo: we stick with the existing strategies we are using to combat rhino poaching. Let’s take a look at how that is working out. The following graph, from the original Quartz piece, shows the annual number of rhinos poached each year in South Africa.


As a percentage of total living rhinos in the world, these numbers are terrifying. There are about 20,000 rhinos in Africa today, according to the most recent counts I could find on Wikipedia (white rhinos black rhinos). If this rate of increase in poaching continues, both the white and black rhino will be driven extinct within our lifetimes. We need to do more than staying the course and hoping for the best. And anyone who wants to object to actual solutions people propose for this problem should base their complaints on sound reasoning and real evidence.

Posted in Uncategorized | Leave a comment

Nigeria is going to be the most important country in the world

I recently came across this article from the Washington Post that presents graphs of population projections through 2100. The writing seems overly pessimistic to me; it has an attitude toward African governance and economic progress that rang true in 1995, but is outdated and incorrect in 2015.

That said, the graphs are great, and fairly surprising. Especially this one:


They are projecting Nigeria’s population to grow to 914 million people by 2100. Even if the truth is just half that figure, Nigeria will draw just about even with the US as the third-most-populous country in the world. Moreover, Nigeria is forecast to be the only country in the top 5 that will be unambiguously growing in population over the current century, making it a source of dynamism and labor supply for the world economy.

Based on my rigorous, in-depth investigation strategy of listening to African pop music and occasionally catching an episode of Big Brother Africa, Nigeria already plays an outsized role in an increasingly salient pan-African culture. The growth of Africa, and the rising importance of Nigeria within Africa (currently one in every seven Africans is Nigerian, a figure that will rise to one in four), mean that its importance is only going to rise.

There will be challenges: for the sake of the environment and the good of its people, the continent needs to urbanize and move away from subsistence agriculture. But the 21st century is shaping up to be an exciting one, and a positive one for Africans in general and Nigerians in particular.

Posted in Uncategorized | 2 Comments

Why do we ever believe in scientific evidence?

Late last night the internet exploded with the revelation that the most influential social science study of 2014 was apparently invented from whole cloth. LaCour and Green (2014) showed that a brief interaction with an openly-gay canvasser was enough to change people’s minds – turning them from opposing gay rights to supporting them. The effects were persistent, and massive in magnitude. This study was a huge deal. It was published in Science. It was featured on This American Life. I told my non-academic friends about it. And, according to an investigation by Broockman, Kalla, and Aronow, as well as a formal retraction by one of the study’s authors, it was all made up. The report by Broockman, Kalla, and Aronow is a compelling and easy read – I strongly recommend it.

The allegation is that Michael LaCour fabricated all of the data in the paper by drawing a random sample from an existing dataset and adding normally-distributed errors to it to generate followup data. I have nothing to add to the question of whether the allegation is true, other than to note that many people are persuaded by the evidence, including Andrew Gelman, Science, and LaCour’s own coauthor, Donald Green.

What I do have to add is some thoughts on why I trust scientists. Laypeople often think that “peer review” means the kind of analysis that Broockman, Kalla, and Aronow did – poring over the data, looking for mistakes and fraud. That isn’t how it works. Referees are unpaid, uncredited volunteers who don’t have time  to look at the raw data themselves. (I have also never been given the raw data to look at when reviewing an article). The scientific method is fundamentally based on trust – we trust that other scientists aren’t outright frauds.* Nothing would work otherwise.

Why do we trust each other? After all, incidents like this one are not unheard of. Speaking from my own experience running field experiments, one important reason is that faking an entire study would be really hard. You’d have got to write grants, or pretend you’re writing grants. Clear your study with the IRB, or sometimes a couple of IRBs. You’d have to spend a significant portion of your life managing data collection that isn’t really happening, and build a huge web of lies around that. And then people are going to want to see what’s up. This American Life reports that LaCour was showing his results to the canvassers he worked with, while the data was coming in (or supposedly coming in). To convincingly pull all of this off, you would basically have to run the entire study, only not collect any actual data.

It is hard to imagine anyone who is realistic with themselves about the incentives they face choosing to go through with all of this. Most scientists don’t get hired by Princeton, don’t make tons of money, don’t get their results blasted across all media for weeks. Most of them work really hard for small payoffs that seem inscrutable to outsiders. The only way to get big payoffs is with huge, sexy results – but huge results invite scrutiny. You might get away with faking small effects that are relatively expected, but if your study gets attention, people are going to start digging into your data. They will try to replicate your findings, and when they can’t they will ask questions. If you do manage to walk the tightrope of faking results and not getting caught, you did a ton of work for nothing.

I can barely conceive of going through all the effort and stress of running a field experiment only to throw all that away and make up the results. I trust scientists to be honest because the only good reason to go into science is because you love doing science, and I think that trust is well-placed.

*Incidentally, this is why I don’t particularly blame Green for not realizing what was up. When I coauthor papers with people, the possibility that they are just making stuff up never even crosses my mind. I am looking for mistakes, sharing ideas, and testing my own ideas and results out – not probing for lies. News accounts show that Green did see the data used for the analysis, just not the underlying dataset or surveys.

Posted in Uncategorized | 1 Comment

Garment workers are people, not props for your viral video

I recently came across this post about a video that raises the question “Who made my clothes?”

The video, started by an organization called Fashion Revolution, suggests an answer: young women like Manisha, who are miserable, and whom you can help by refusing to by the t-shirts they make and instead donate.*

But wait a second – who did make my clothes? Specifically, who are the people in the video who (it is suggested) made the t-shirts being sold? The only people with any agency in the video are the westerners who are choosing not to buy the shirts. The garment workers appear only in still photos in which they appear harrowed and fearful. They don’t do or say anything.

Who is Manisha? Why does she work in this factory? Does she support the idea of consumers refusing to buy the clothes she is paid to make? She doesn’t say anything in the video, and if Fashion Revolution gives her a voice or an identity at all, they don’t make it easy to find on their website:

I looked through the site’s blog, and didn’t see anything written by employees in the garment factories featured in the video. There are some quotes from people employed by garment makers that Fashion Revolution deems socially conscious, but nothing about the folks whose lives we are ostensibly trying to change. They appear only as a way to manipulate viewers of the videos. Fashion Revolution shows them to us – I want to know who they are and what they think about this campaign. I want to know why they work in these jobs, and what they would do instead if the jobs didn’t exist.

Fortunately, there is a way to learn about those questions. Planet Money conducted an epic 8-part investigation into how some t-shirts they bought were made (I previously mentioned this series in  this post). In the episode “Two Sisters, a Small Room and the World Behind a T-Shirt” the producers learn that two of the garment workers who produced their shirts are a pair of sisters from Bangladesh named Shumi and Minu – and then they travel to Bangladesh, meet Shumi and Minu, and talk to them about their lives and their careers. Bad work conditions do come up, but so do many other problems and concerns and joys and triumphs in their lives. The followup to “who made my clothes?” shouldn’t be “let’s stop buying them” but “what do the human beings who made my clothes see as their problems, and how can we help?”

*There is no evidence from Fashion Revolution’s donation page that the money is actually going to Manisha or any of the other garment workers in the video. Instead, it sounds like the donations are going to raising awareness about the issue. That’s probably fine, but do the donors in the video understand that? Or do they think they are directly helping Manisha?

Posted in Uncategorized | 1 Comment

Is it impossible to prevent usury?

Usury laws are intended to prevent creditors from charging predatory interest rates. While morally pleasing, their practical effects are debatable, since borrowers whose credit profiles would call for ultra-high interest rates might not have access to credit at all in the presence of such laws.

But do they even work? A beautiful new paper by Brian Melzer and Aaron Schroeder says they do not, if sellers are allowed to offer loans directly:

We study the effects of usury limits on the market for auto loans and find little evidence of credit rationing. We show instead that loan contracting and the organization of the loan market adjust to facilitate loans to risky borrowers. When usury restrictions bind, auto dealers finance their customers’ purchases and raise the vehicle sales price (and loan amount) relative to the value of the underlying collateral. By doing so, they arrange loans with similar monthly payments and compensate for credit risk through the mark-up on the product sale rather than the loan interest rate.

Unless we are willing to ban auto dealers from offering loans – which I suspect would be difficult – then usury protection effectively does not exist in the auto loan market, even though usury is banned by law.

This is an example of why economics is such a compelling subject: even seemingly straightforward solutions to simple problems may completely fail to work. Human beings are extremely clever and complex animals, and it is quite tough to design systems to shape their behavior in the way we want to.

Posted in Uncategorized | Leave a comment

Focal points and minibuses

It’s 1975. You’re on vacation in New York City, and you’ve made plans to meet an old friend, also on vacation there, tomorrow. The problem is that you don’t have their phone number or any other way to get in touch with them, and you didn’t agree on a place or time to meet. Where do you go to meet your friend – and at what time?

The above is one of my favorite game theory problems, because it is one that even non-experts can solve. It is interesting in that there are many “correct” answers – any place in the city, and any time, will work, as long as both you and your friend make the same choice. But what time and place will maximize the chances of that happening? I’ve thrown out the problem in a number of casual conversations, and the modal answer is 12:00 noon in front of the clock at Grand Central. Since the essence of the game is to guess what the other person will do, this is also the right answer in that it maximizes the chance that you and your friend will win – the “focal point” of the game. It’s also the answer that Tom Schelling got when he tried this problem out back when he invented it.

I was reminded of this problem by a recent post on what is easily my favorite new development blog, NPR’s Goats and Soda, that covered the reversal of the Kenyan government’s ban on decorating minibuses. A bit of background: minibuses, which go by various names in different countries, are the chief means of public transit in most of eastern and southern Africa. They work along fixed routes, and often for fixed fares (in Malawi it is technically illegal to raise prices unless a government board approves a fare hike). A minibus will typically sit at the origin point for its route until it is totally full, and then head to the end point.

Commuters are thus faced with a problem: there are lots of minibuses at the bus stop. I want to take the one that leaves soonest since it will arrive sooner. Which one do I board? NPR says it is the one with the most garish decorations, because “if young people prefer a painted bus that plays loud music, and they seem to, then it fills up faster”. That is the right answer – but the wrong reason! I would bet money that the painted bus will fill up faster even if there are no young people around or they don’t prefer it, because it is an obvious focal point for the game.

Minibus operators are very aware of this focal point problem, and employ multiple techniques to solve it. One approach is collective action: some bus stops have an enforced queue of buses*, so customers can only board the first one in the line. Another one that I particularly like involves faux customers called “timing boys” who board the bus and sit on it until it is nearly full, then get off (usually to a hail of jeers from the actual commuters). Since a fuller bus is more attractive (and anyway an obvious focal point even if customers are aware of the ruse), this attracts customers quicker and leads to faster departures. Policymakers should keep in mind that when a minibus owner gets “CNN BREAKING NEWS” spraypainted onto his bus, it’s more than just for fashion – it makes the transit network more efficient.

*In Malawi these queues are often enforced by cabals of stationary bandits known as “touts”, who impose an effective tax on the bus operators.

Posted in Uncategorized | Leave a comment