Suppose you want to construct a democracy index measuring the level or lack of democracy in different countries in the world. The normal thing to do is to select some supposedly essential characteristics or attributes of democracy and try to measure the level or presence of those. So, for example, you may select free speech, elections, judicial independence and a number of other characteristics. Some of those are perhaps already measured and you can simply take those measurements. For others, you may have to set up your own measurement (e.g. a survey, analysis of newspapers or official documents etc.), or use a proxy.
In any case, you’ll end up with different datasets on different attributes of democracy, and you’ll have to bring those datasets together somehow in order to make your overall index, you single country-level democracy score. The problem is that the datasets contain different kinds of scales which cannot as such be aggregated into a global index. The scales and the values in the scales have to be normalized, i.e. translated into a common metric.
normalized value = raw value/maximum raw value
First, however, you have to rescale some existing scales so that they start at 0 – in other words, so that the lowest score is 0 (instead of starting at 1 for example, or at -10 such as the Polity IV scale). This way, all scales will have a normalized range from 0 to 1; 0 being the negation or total absence of the attribute; 1 being the complete and perfect protection or presence of the attribute.
What about weighting the different attributes? Some may be more important for a democracy than others. However, introducing weights in this way inevitably means introducing value judgments. While value judgments can’t be avoided (they’ll pop up at the moment of the selection of the attributes as well, for example), they can be minimized. If you choose not to use weighting, you consider all attributes to be equally important, which is a view that can be defended given the often interdependent nature of the attributes of democracy (an independent judiciary for example will likely not survive without a free press).
Once the different data sources are translated into normalized scales and, if necessary, weighted appropriately, they have to be aggregated in order to calculate the global index of quality of democracy. One possible aggregation rule would be this:
global index = source 1 * source 2 * ... * source n.
So a simple multiplication. But that would mean that a value of 0 for one attribute results in labeling the country as a whole as having 0 democratic quality. This is counter-intuitive, even with the assumption of equal importance of all attributes. Hence, a better aggregation rule is the geometric or arithmetic mean (or perhaps the median).
However, there’s also a problem with averages: low scores on one attribute can be compensated by high scores on another. So very different democracies can have the same score. Also, within one country, a high score on suffrage rights but 0 on actual participation would give a medium democracy score, whereas in reality we wouldn’t want to call this country democratic at all (the score should be 0 or close to 0). Perhaps we can’t avoid weights after all.
More posts in this series are here.