Global Burden of Disease Estimates: Secret Recipes or Spoiled Ingredients?

October 16, 2013

By Victoria Fan, APHRC – CGD Data for African Development Working Group Member and CGD Research Fellow

*This blog was originally posted on the Center for Global Development’s Health Policy Blog.View the original post here. Victoria Fan is a research fellow at CGD and a member of theAPHRC – CGD Data for African Development Working Group.

Although counting the sick and dead in a country can seem quite dull if not morbid, these facts are critical inputs to designing any national health policy, let alone global priorities in health. Yet 85% of the world’s population still lack systems that register births and deaths along with high-quality data on causes of death.

The Global Burden of Disease (GBD) – whose first edition was commissioned by the World Bank in 1991 and whose latest edition came out in December 2012 in the Lancet – was the first systematic attempt to count the sick and dead in a rigorous way. The GBD researchers used all data sources available to them. And while this work is a landmark in global-health history and deserves praise, the underlying data the researchers use are of poor quality. For example, it’s hard to figure out how many deaths were actually counted in the latest GBD, and how many deaths were “extrapolated” from a variety of methods. What’s worse, there has been slow – if any – progress in improving the underlying data since the first GBD two decades ago.

These poor ‘raw ingredients’, the underlying raw data, are the main reason why there is such uncertainty and inaccuracy in many global-health statistics. Even with the wide application of new statistical methodologies by top-notch researchers, only so much can be done in correcting for biased and missing data. Or, put another way, even the best recipes and best chefs in the world can’t make a meal out of spoiled (or non-existent) ingredients.

There are countless examples where global estimates were significantly revised because of these uncertainties. The most recent are the two-fold differential between estimates by WHO and IHME in the number of malaria deaths. Another previous big “guess” was also for HIV/AIDS – when in 2007 UNAIDS decreased its estimate of people living with HIV by over 6 million to 33.2 million. Maternity mortality estimates may have been previouslyoverestimated, too. The list of embarrassing “whoops, bad guess” goes on, and those “whoops” will keep on happening under the current business-as-usual scenario.

Meanwhile, people have been criticizing the Global Burden of Disease, particularly for its lack of transparency and lack of consultation with countries (see here). The GBD authors recently gave an entirely predictable explanation to their actions – the academics owning the data need to publish.  As a researcher, I can sympathize. And while there is reason for concern about the academic replicability and transparency of the work by folks in Seattle or Geneva, these issues don’t strike to the real heart of the problem:  The problem is not with the chefs or their secret recipes. It’s with the bad ingredients.

During a recent meeting in Geneva on the state of global-health statistics, Dr Richard Horton, editor of the eminent Lancet journal, highlighted this central problem through his always-fascinating stream of tweets (view his tweets here).

Horton concluded his stream of tweets with a snap-shot (see below) of the “Recommendations on the way forward” produced from this meeting, with notable emphasis on strengthening country health information systems and country capacity. But at first glance, these recommendations do not learn from the failures of making progress on country statistical capacity over the past two decades. The recommendations need clarification on what exactly “strengthening” of health information systems and country capacity means, or why it hasn’t already happened over the past twenty years. Recommendations made by academics and policymakers without extensive leadership from countries run the risk of being ineffective.

Which is why I’m encouraged by the Data for African Development working groupconvened by Alex Ezeh of the African Population and Health Research Center and Amanda Glassman of the Center for Global Development. The working group has convened a number of local, country actors particularly connected with the Ministries of Statistics, and National Statistics Offices, along with donors. The group has focused on the poorly aligned incentives to collect this data at all levels in their creation, along with the political economy and institutional arrangements that have helped or hindered better statistical capacity. Their perspectives, I believe, will shed new light and offer new recommendations in addressing these persistent problems.

The author thanks Alex Ezeh, Amanda Glassman, and Jenny Ottenhoff for excellent comments.