By Karthik Dinne
Why the relationship between factors must be considered when evaluating the relationship between one factor and the outcome
In an earlier post, we discussed the dangers of conflating an evaluation of a product with an evaluation of its theme. In this post, we will discuss the causal framework used to interpret evaluations. Most caveats about evaluations don’t deal with their design but more with the communication and interpretation of their results.
In a typical evaluation, the outcome of interest is often dependent on multiple factors. Evaluations can be broadly classified into two categories based on the manner in which these factors are dealt with.
- Evaluations of programs which span across multiple factors: Do cash transfers increase the learning outcomes of children?
- Evaluations of the individual factors: Does providing textbooks to children improve learning outcomes?
In the first category, the cash transfers aren’t a part of education per say; they indirectly influence education by affecting the components of education delivery. One such component is the provision of textbooks as evaluated in the second category. In order to conduct the first category of evaluations it is often necessary to isolate each factor by conducting the second category of evaluations for each of them. This post discusses one of the caveats in interpreting evaluations of an individual factor where the outcome is determined by multiple factors.
To continue with the example of education, let us assume that the individual factors affecting learning outcomes are limited to:
- Providing free textbooks
- Increasing teacher attendance
- Increasing access
- Teacher training
- De worming
- Decreasing Pupil Teacher Ratio
After looking at them, most people would probably view these factors using the following model:
In this framework, each factor is only examined in relationship to the eventual outcome. A change in the factor is considered sufficient to impact the net outcome, even if this is not necessarily so. Such an interpretation means that the net outcome is viewed as the sum total of the impacts of each factor. It would thus be possible to achieve increased learning outcomes by only affecting one factor, e.g. providing more textbooks, or improving access to schools, and so on.
The reality, however, is more like this:
Factors rarely exist in isolation and are often interdependent on one another; they are all necessary conditions to achieve the outcome. It is similar to an electric circuit: breaking the circuit at any point will affect the flow of electricity, even if the circuit is otherwise complete. In this framework, improving learning outcomes would require all the factors to be pursued simultaneously. Providing more textbooks would have to be accompanied by improvements in teacher attendance and training, de-worming measures, etc.. It is only until the threshold value for each factor is reached that we start seeing a change in the outcomes.
The danger is in interpreting evaluations of individual factors using the first model when the relationship between the factors more closely resembles the second framework. It is possible to arrive at conclusions that are unreliable before even the question of external validity can be asked. The intervention administered on the factor may have failed to affect the final outcome only because the other factors remained unchanged. If the intervention was about increasing access to schools, we might conclude that increasing access won’t result in an improvement of learning outcomes. This might cause us to ignore the fact that improving access will only affect learning outcomes if accompanied by increased teacher attendance. This interpretation would have serious consequences on educational policy; it would probably lead to budget cuts for measures improving access to schools even though such measures might actually be vital.
It is therefore necessary to not view each factor in isolation when evaluating its relationship to the outcome. It is possible to misattribute its failure to affect the outcome. While it may be necessary to isolate each factor for greater understanding, their position in the whole picture must always be kept in mind. The interdependence between factors calls for a lot of care when interpreting evaluations of them, especially when the interpretations can decide policy priorities.
Karthik Dinne is an alumnus of the Graduate Certificate of Public Policy and can be found on Twitter on his handle @dkarthik. The views expressed here are personal.