This is Science Fictions with Stuart Ritchie, a subscriber-only newsletter from i. If you’d like to get this direct to your inbox, every single week, you can sign up here.

Yesterday brought some very positive-sounding news, from the scientific journal Nature: cash transfers make people live longer.

That is, simply giving money to people in low- and middle-income countries – either with some condition attached, like ensuring they attend school or a particular course, or purely unconditionally – protects them from the rates of early mortality that are so common in poorer countries.

This seems highly plausible to me. I’m very positively disposed to these kinds of cash transfers: especially with those that are unconditional, they allow people to pursue whatever things they want with complete freedom, rather than having governments or NGO workers breathing down their necks and trying to micromanage their lives.

And the previous evidence on the effects of cash transfer programmes certainly looks positive as a whole. A 2016 report by the Overseas Development Institute talked about the “clear and significant impacts” of cash transfers on things such as children’s school attendance and on savings and investment, arguing that “not only can cash transfers play a role in reducing poverty by redistributing resources to the poor, they can also foster their economic autonomy and self-sufficiency”.

That report did, however, note that the medium- and longer-term effects of cash transfers were less well known, and that health was one of the areas where not enough studies had been done.

That’s where this new Nature paper comes in.

The Nature study

The first thing to note about the new Nature paper is that it isn’t itself a new experiment on cash transfers: it’s a “secondary” analysis. That is, they found all the low- and middle-income countries where any kind of cash transfer had been done (37 countries in total) – and included cash transfers that were conditional or unconditional, involved just a particular group of people or the entire population, and were targeted at different age groups. They then matched up all this cash-transfer data with data on mortality rates from the same countries at the same time as the transfer.

One way of looking at this is as a clever, creative, resourceful way of aggregating different data sources to answer an important question. A perhaps more jaded view is that, in stitching together different data sources, there’s a lot of room for statistical noise. None of the cash-transfers was set up as a study specifically to measure mortality rates, so there’s always going to be a lot of uncertainty around this kind of secondary analysis.

In any case, to join up the two kinds of data, the researchers used a “difference-in-differences” analysis, where they looked at the mortality rates in people who were and who weren’t given a cash transfer, both before and after the cash transfer actually happened. It’s a common kind of analysis in econometrics, and in certain circumstances it can allow you to (tentatively!) draw a causal conclusion from purely observational data.

And here’s how they summarise their results: “We found that cash-transfer programmes were associated with a 20 per cent reduced risk of death in adult women and a 8 per cent reduced risk in children aged younger than five years old.”

As always, here’s the “but”

Who could argue with that excellent-sounding result? Except… this is actually the sort of statement that should ring alarm bells.

It seems like the results only apply to particular subsets of the data. And indeed, when you look at the overall graph of results, you can see that the results were split into five groups: women, men, children under five, children aged five to nine, and children aged 10 to 17.

The result for women looks fairly convincing, but that smaller 8 per cent effect for children under five is right on the very borderline of what would be considered “statistically significant” (for statistics fans, the 95 per cent confidence interval goes right up to 0.99, where it would count as not statistically significant if it touched 1.00). This means that it’s probably quite fragile: you could easily imagine it being different with a slightly different dataset, or a slightly different age split.

Indeed, the reasons for choosing these specific age splits (under five, five to nine, 10-17) – and for not analysing the adult data separately by age, since they did so for the children – aren’t explained. And here’s the key problem: there’s no pre-analysis pre-registration of the plans for the study, so we can’t know whether the data were always planned to be split this way, or whether the authors perhaps analysed other specifications of the data.

Later in the study, the researchers run lots of other analyses: on conditional versus unconditional transfers (no differences in their effect on mortality), on whether the women affected died in pregnancy or in other situations (effects of the cash transfer on both), and on the region in which the cash transfer happened (cash transfers seemed to work better in countries with little health infrastructure), among other things.

The problem here, of course, is one of “multiple comparisons”: there are a lot of variables analysed, and the more analyses you run, the more likely you are to find spurious, false-positive effects (I’ve talked about this issue before with regard to, for example, the studies on the effects of social media). Several of the results were statistically significant; many weren’t. That’s a classic red flag for false-positives.

Instead of doing any kind of statistical multiple-comparisons correction though, the researchers just say these extra results “should be considered exploratory in the setting of multiple comparisons”. That’s fair enough to some extent – the reader can always make up their own mind about which results to accept and which to doubt.

But when writing up their results in other contexts, they were less circumspect, not mentioning this issue at all. Moreover, it’s not clear why these specific results should be considered the exploratory ones, since usually “exploratory” analyses are the ones not included in a pre-registration. In this case, there’s no pre-registration at all, so we just have to take the authors’ word for it that their “exploratory” analyses were always planned to be the less important ones.

A missed opportunity

It’s a crying shame, really. One simple act – pre-registering the study and making a clear, concrete, public plan for how they’d run their analysis – would’ve dramatically improved the believability of these results. As it is, the authors just weren’t transparent enough about the way they did their study for us to have real confidence in the results.

As I said above, I’m a big fan of these cash-transfer programmes, and they seem like a very plausible way of reducing poverty and improving people’s lives. But do they protect against early mortality? The sad thing is that, even after reading an article in Nature – the world’s “top” scientific journal, where you’d expect strong, definitive results – I don’t know.

Other things I’ve written recently

How much damage to the climate will these dogs do? (Photo: David Parry/PA)

Did you see the recent claim by the boss of a private jet company that owning three dogs produces the same amount of CO2 as flying private? I tried to track down the numbers.

It’s been a bumper week for AI: I wrote this article on the hyped study from last week about how AI has helped us discover a new antibiotic, and this one giving a few more plausible (though perhaps not all that probable) scenarios for AI doom.

Science link of the week

I thought this was a good article on “misinformation” by Matt Yglesias. The term has started to become synonymous with the kind of bad information you get on the right wing of politics: these days in particular it focuses on false claims about the “dangers” of the Covid vaccines. That’s a serious and very prominent concern (one I’ve recently written about). But Yglesias rightly reminds us that there’s misinformation all across the political spectrum. 

This is Science Fictions with Stuart Ritchie, a subscriber-only newsletter from i. If you’d like to get this direct to your inbox, every single week, you can sign up here.

By admin