How Much Can We Generalize from Impact Evaluations? - Working Paper

For those of us logarithmically challenged, Vivalt's "How Much Can We Generalize from Impact Evaluations?" is a daunting task. The topic is quite relevant, the use of impact evaluations (and specifically Randomized Control Trials as the 'golden standard') have been growing in the evaluation of development interventions. However, a common counter-argument has been on the ability to generalize the results to other contexts. This paper does try to address that question, unfortunately (or maybe expectedly given the trend of mathematical jargon in social sciences) the paper is buried under equations, tables, regressions and the like. Meaning that a topic of great importance for development practitioners gets lost in the medium. I wouldn't be surprised if many of my colleagues give up just with the abstract: 

"Impact evaluations aim to predict the future, but they are rooted in particular contexts and results may not generalize across settings. I founded an organization to systematically collect and synthesize impact evaluations results on a wide variety of interventions in development. These data allow me to answer this and other questions across a wide variety of interventions. I examine whether results predict each other and whether variance in results can be explained by program characteristics, such as who is implementing them, where they are being implemented, the scale of the program, and what methods are used.  I find that when regressing an estimate on the hierarchical Bayesian meta-analysis result formed from all other studies on the same intervention-outcome combination, the result is significant with a coefficient  of 0.6-0.7, though the R-squared is very low.  The program implementer is the main source of heterogeneity in results, with government-implemented programs faring worse than and being poorly predicted by the smaller studies typically implemented by academic/NGO research teams, even controlling for sample size.  I then turn to examine specification searching and publication bias, issues which could affect generalizability and are also important for research credibility.  I demonstrate that these biases are quite small; nevertheless, to address them, I discuss a mathematical correction that could be applied before showing that randomized control trials (RCTs) are less prone to this type of bias and exploiting them as a robustness check." 

Those brave (or foolish) enough to have gone on would probably take away a couple of points:

- Academic/NGO scalability into government programmes is not a given. Taking into account that many of our programmes, specially those with governance components, make claims of eventual government take over/transfer (a favorite explanation for the sustainability section of proposals), an even more nuanced view is needed for future impact. Alternatively, further study is needed to find out how consistency of impact can be ensured.

- Heterogeneity. A fancy word for 'results all over the place'. Not only because of programme implementer as see earlier, but even within studies there is a high degree of variation. I believe this points out to the fact that we may not know what the underlying factors that affect the intervention are (making the control of variables much harder). Also the general underpowering (statistical) of studies, sample sizes may not be large enough to smooth out results (this is a hunch, as I am in no position to prove it or debate it), and, of course, context.

- Impact evaluations do have some predictive power in development. We would be in trouble if the result would have been negative! 

¿Qué hay de nuevo Yemen?

El primer ministro Khalid Bahah anuncio el sábado la composición (parcial) del gabinete. De 34 puestos, nueve irían al General People’s Congress (GPC), nueve al Joint Meeting Parties (JMP), seis al Movimiento del Sur y seis para los Houthi. Los ministerios de asuntos exteriores, defensa, interior y finanzas serán asignados directamente por el presidente Hadi. El JMP ha protestado por la distribución de ministerios y no está claro hasta qué punto colaboraran en la formación de gobierno.
 
Los incidentes violentos (incluyendo ataques suicidas) continúan en varios lugares. Milicia Houthi capturo el castillo de Al-Hodeida tras combates con Hirak Al-Tuhami (movimiento basado en zona oeste del país). Houthi y Al-Qaeda/aliados tribales se han enfrentado en Ibb y Al Bayda (alrededor de la ciudad de Rada)

Protestas y manifestaciones pidiendo la independencia continúan en las provincias del sur. El Movimiento del Sur ha fusionado sus consejos en uno, el Consejo Supremo del Revolucionario y Pacifico Movimiento del Sur.

Non-Farm Enterprises in Rural Africa: New Empirical Evidence - Working Paper

Probably the biggest takeaway from Nagler and Naude's "Non-Farm Enterprises in Rural Africa: New Empirical Evidence" is the recognition that 'Africa is not a country', or as they put it: Heterogeneity. They compare data from six countries and data diverges significantly as dynamics are tied to local context, not only at national level but also at sub-national. Nevertheless, the issue of non-farm enterprises remains key in many African (and other) countries, and this paper highlights it quite well.

 

Abstract:     

"Although non-farm enterprises are ubiquitous in rural Sub-Saharan Africa, little is yet known about them. The motivation for households to operate enterprises, how productive they are, and why they exit the market are neglected questions. Drawing on the Living Standards Measurement Study -- Integrated Surveys on Agriculture and using discrete choice, selection model and panel data estimators, this paper provide answers using data from Ethiopia, Niger, Nigeria, Malawi, Tanzania, and Uganda. The necessity to cope following shocks, seasonality in agriculture, and household size can push rural households into operating a non-farm enterprise. Households are also pulled into entrepreneurship to exploit opportunities. Access to credit and markets, household wealth, and the education and age of the household head are positively associated with the likelihood of operating an enterprise. The characteristics are also associated with the type of business activity a household operates. Rural and female-headed enterprises and enterprises with young enterprise owners are less productive than urban and male-owned enterprises and enterprises with older owners. Shocks have a negative association with enterprise operation and productivity and a large share of rural enterprises does not operate continuously over a year. Enterprises cease operations because of low profits, a lack of finance, or the effects of idiosyncratic shocks. Overall the findings are indicative that rural enterprises are "small businesses in a big continent" where large distances, rural isolation, low population density, and farming risks limit productivity and growth."

Some points to note from the study:

- entrepreneurship have had a very urban focus, and when it comes to the rural areas the focus is on agriculture. The study show the importance of non-farm enterprises in smothing shocks and providing additional income

- Most livelihoods and entrepreneurship interventions focus on the individual. However,
in this context, enterprises are a household activity (both in decision making and operation), specially taking into account distribution of labor according to seasonality (planting and harvest), number of available workers and household tasks (and gender-based allocations).

- Response to shocks (drought, price, conflict) is a major reason for enterprise creation, however not all shocks have the same effect on the choice of sector. Many differences are country specific (different responses to the same kind of shock). Moreover, productivity of enterprises born out of necessity (shock response) is lower than those out of choice. This may help us rethink the programming 'mix' regarding both prevention and response to crisis.

- In some cases, education was positively correlated with entrepreneurship, probably due to the fact that educated individual do not have a comparative advantage in a rural setting where waged employment is limited and therefore reallocate into self-employment. The role of education (or the kind of) in rural settings have to be thought further.

- Each non-farm sector has its own profile. The variables studied (education, number of adults in the household, age and credit access) have different effect depending on the sector: agribusiness, sales, trade, restaurant, professional services, etc...

- Distance to population centers. Highlighting the importance of rural secondary towns and cities, as enterprises located up to 10 km from a population center are the most productive (on average urban enterprises are more productive than rural). Productivity drops significantly when they are more than 50 km away. Large gains in poverty reduction have actually been linked to secondary cities rather than major cities or capitals.

-Youth. Younger households are not only less productive but also less probable to engage in non-farm entrepreneurship in comparison with middle age groups. To note this is talking about household heads, and not necessarily individual engagement in activity.

- Seasonality and enterprise life. Agriculture remains the key driver in many rural settings and non-farm enterprises adapt to that cycle. Enterprises, in many cases, appear/grow and disappear /decrease according to the planting and harvesting needs, and not necessarily seen as permanent activities

- Exit reasons. The most self-reported reasons of enterprise termination are low profitability, lack of finance, unreliable supplies, as well as the impact of  death or illness in
the family (specially in settings with limited insurance or social protection)

While this paper provides and overview might be good for getting ideas where to look further, we would be better off going into country specific studies when it comes to programming and policy recommendations/development.

Interesting links

Latest round of interesting articles:

- Why are you armed? Gun culture in Yemen

- Sun Tzu, entrepreneurship and the Austrian School (economics) in "The Political Economy of 'The Art of War'"

- Genetic testing in Rapa Nui (Easter Island): Polynesian, Native American and European contacts.

Do Traffic Tickets Reduce Motor Vehicle Accidents? - Working Paper

Traffic tickets are very unpopular among drivers, but to what extent do they affect behavior? do they reduce fatalities? These questions are addressed by Luca's paper "Do Traffic Tickets Reduce Motor Vehicle Accidents? Evidence from a Natural Experiment".

 
Abstract:     

"This paper analyzes the effect of traffic tickets on motor vehicle accidents. OLS estimates may be upward-biased because police officers tend to focus on areas and periods with heavy traffic and thus higher rates of accidents. This paper exploits the dramatic increase in tickets during the Click-it-or-Ticket campaign to identify the causal impact of tickets on accidents using data from Massachusetts. I find that tickets significantly reduce accidents and non-fatal injuries. However, there is limited evidence that tickets lead to fewer fatalities. I provide suggestive evidence that tickets have a larger impact at night and on female drivers."

 

The deterrence effect of law enforcement is a much debated issue, and traffic tickets are even more controversial on their own as they are/could be used as revenue source. Nevertheless, the stated purpose of the fines is to enforce traffic regulation and, if we are to believe the ad campaigns, prevent fatalities. Using a seat-belt law and enforcement campaign is a nice natural experiment, as seat-belt are designed for preservation of life.

The study shows that tickets do lead to a reduce number of accidents but not necessarily to a decrease in fatalities. This may be due to the allocation of police resources, that may be placed in locations and times that may not prevent fatal accidents (i.e. nearly 50% of fatalities occur at night).

¿Qué hay de nuevo Yemen?

El Comité encargado de preparar el borrador de la nueva Constitución de Yemen dice haber completado el 80%, desafortunadamente la parte restante también es la más espinosa con temas como federalismo o inmunidad de altos cargos. Se espera un primer borrador para noviembre, y el comité se encuentra en Abu Dabi.

El Consejo de Seguridad de Naciones Unidas podría imponer sanciones a prominentes yemenís acusados de fomentar la inestabilidad. Aunque la amenaza se ha hecho múltiples veces en el pasado, eventos recientes no hace sino aumentar las posibilidades. Entre la lista de sancionables esta el ex presidente Saleh (acusado de colaborar con los Houthi), su hijo Ahmed Ali (general a cargo de la Guardia Republicana antes de ser nombrado embajador en EAU), el líder Houthi Abdulmalik al-Houthi, su hermano Abdulkhaleq al-Houthi y el líder militar Houthi Abu Ali al-Hakem.

El movimiento Hirak ha establecido campamentos de protesta pidiendo la secesión tanto en Aden como en Mukalla (Hadramout). El movimiento independista está cogiendo fuerza, especialmente en reacción al avance Houthi, y han pedido al gobierno que retire a las tropas y administración del Sur para finales de mes... algo improbable, ya que significaría la desaparición del gobierno a efectos prácticos. El movimiento Tehama (costa occidental de Yemen) esta organizando manifestaciones pidiendo la retirada Houthi de Al-Hodeyda y la región.

La expansión Houthi empieza a encontrar más resistencia. Tribus locales en Ibb están expulsando o capturando militantes Houthi, y, en otras partes de la provincia, Al Qaeda ha tomado múltiples poblaciones tras combates con los Houthi. No se sabe hasta que punto hay colaboración entre la tribus locales y Al Qaeda, pero no es descartable que sean aliados de conveniencia. Los combates continúan en Al Bayda, principalmente alrededor de la ciudad de Rada.

El oleoducto en Marib fue saboteado por militantes tribales, provocando un paro temporal en el bombeo.

Interesting links

Latest round of interesting articles:

- How to talk to terrorist or rather why we should.

- The evidence on travel bans for diseases like Ebola is clear: they don't work

-Taking inspiration from the Torah, "Megillath Esther and the Rule of Law: Disobedience and Obligation"

-A Development Agenda without Developing Countries? The Politics of Penurious Poverty Lines, by Lant Pritchett

Investigating El Niño‐Southern Oscillation and Society Relationships - Working Paper

"Investigating El Niño‐Southern Oscillation and Society Relationships" by Zebiak et al, looks into climate forecasting on three sectors: water, agriculture and health. It shows the role that forecasting can have as part of planning and response in areas that have large societal impact (beyond the extreme weather events associated in the media with El Niño or La Niña). Abstract:  

"Throughout at least the past several centuries, El Niño‐Southern Oscillation (ENSO) has played a significant role in human response to climate. Over time, increased attention on ENSO has led to a better understanding of both the physical mechanisms, and the environmental and societal consequences of the phenomenon. The prospects for seasonal climate forecasting emerged from ENSO studies, and were first pursued in ENSO studies. In this paper, we review ENSO's impact on society, specifically with regard to agriculture, water, and health; we also explore the extent to which ENSO‐related forecasts are used to inform decision making in these sectors. We find that there are significant differences in the uptake of forecasts across sectors, with the highest use in agriculture, intermediate use in water resources management, and the lowest in health. Forecast use is low in areas where ENSO linkages to climate are weak, but the strength of this linkage alone does not guarantee use. Moreover, the differential use of ENSO forecasts by sector shows the critical role of institutions that work at the boundary between science and society. In a long‐term iterative process requiring continual maintenance, these organizations serve to enhance the salience, credibility, and legitimacy of forecasts and related climate services."

ENSO studies started in the 80's with an important growth in the 90's that led to a better understanding of its consequences and also the development of forecasting tools. How have these tools translated into improved decision-making? By looking at three sectors that are affected by ENSO in a wider geographical area (from the Indian Ocean to the American continent, the authors show the different societal intake of forecasting tools.

Sectors that have previously used weather forecasting (like agriculture and water management) have been the ones that adapted ENSO modelling the most. Previous knowledge of similar tools allowed for users to be comfortable with the models and were conceptually more ready. Entering forecasting in sector that have not previously used it may require a tailored approach with simple quick wins over a longer period of time rather than complex models.

Institutions matter (a much repeated point in development literature), not only their presence (by using already established networks to distribute knowledge like agricultural extension) but also their design (focus on policy, research, awareness raising, best practice implementation, centralized vs decentralized, etc...). Tools have to be seen as relevant to the user, institutional design tell us not only the mandate or objectives but, maybe more importantly, who the user actually is.

Linked to the previous point, forecasts are a decision-making tools should help those making decisions. These seems obvious, but we have to remember that in many instances the users of forecasts are not the final beneficiaries of a decision. For example, a water management board makes decisions that affect water users by managing water flows. The forecasts itself will have to compete with other dynamics within the institution in order to be effective. On the other hand, in the case of agriculture, the farmer may be the recipient of the forecast, the decision-maker and the final beneficiary. That shortened decision-making process may help to explain why it is in agriculture where the uptake of ENSO forecasting tools has been the strongest as the incentives are aligned.

 

Whether we are interested in climate events or not, this paper does point out interesting issues on the usage of forecasting in different settings and sectors, and present lessons learnt that could be transferred to other sectors. Maybe the title is over-ambitious with the "society relationships" part.  

Keep the Kids Inside: Juvenile Curfews, Bad Weather, and Urban Gun Violence - Working Paper

Carr and Doleac make an interesting comparison between imposed curfew and 'natural' curfew in "Keep the Kids Inside: Juvenile Curfews, Bad Weather, and Urban Gun Violence". 

Abstract:     

"Gun violence is an important problem in many American cities, large and small. Due to limited data, it has been difficult to convincingly test the impacts of government policies on the quantity and geography of gunfire. This paper is the first to use a new source of data on gunfire incidents, which does not suffer from selective underreporting common in other crime datasets. We test the incapacitation effects of two interventions in Washington, DC: (1) juvenile curfews, and (2) rain. Both work primarily by keeping would-be offenders indoors. The former is a common, but extremely controversial, policy used in cities across the United States, and its impact is highly sensitive to how it is enforced. The latter is an intervention over which we have no control, but can be thought of as a perfectly-enforced incapacitation "policy": anyone who stays outside during a rainstorm gets wet. We use exogenous variation in the hours affected by each intervention to estimate its causal impact on gun violence and reported crime. We find minimal evidence that juvenile curfews are effective, but rainstorms result in large, statistically-significant reductions in gun violence and other crime. It thus appears that it is possible to remove would-be offenders from the streets, but juvenile curfews do not have this effect. We interpret these results as evidence that incapacitation works as a crime-prevention tool, and a reminder that implementation and enforcement are key determinants of a policy’s success".

The study serves as a proof-of-concept for 'incapacitation' while at the same time showing that one of the tools used, juvenile curfew, is not really effective. Policy implications are interesting. Is rain-making technology a useful tool during crime waves? Seeing that juvenile curfew is not effective, which is the key failure of the curfew? what other tools should we be looking at in terms of incapacitation? Not to forget that incapacitation could also be understood from a positive perspective, that is kids are not on the streets because they are doing something else.

Radicalisation and al-Shabaab recruitment in Somalia - Paper

Plenty of ink has been spilled about Al-Shabab and the Somali context (not that we notice much difference most of the time); Botha and Abdile's "Radicalisation and al-Shabaab recruitment in Somalia" look at one key organizational development component: Recruitment. Summary below.

"Effective counter-radicalisation strategies should be based on an empirical understanding of why people join terrorist organisations. Researchers interviewed former al-Shabaab fighters and identified a complex array of reasons for why they joined the organisation. Interviewers developed a profile of typical al-Shabaab recruits and identified factors facilitating their recruitment, including religious identity, socioeconomic circumstances (education, unemployment), political circumstances and the need for a collective identity and a sense of belonging. The reasons for al-Shabaab’s rise are discussed and recommendations are made to the Somali government, countries in the region and international organisations and donors on how to counter radicalisation and recruitment to al-Shabaab."

The study doesn't really offer a novel insight into Al-Shabab's recruitment, it is a much needed reminder that many core issues that fuel radicalization are still ongoing. 88 former fighters were interviewed (plus 7 off-the-record), therefore it remains to be seen if the fact that they are not fighting anymore may give us a biased sample. Nevertheless it is a valuable attempt. However, I would have preferred to include a "control" sample of Somalis that did not join Al-Shabab, this way we can compare if the reasons for joining are unique for this group or shared across Somalia: i.e. if economic reasons are given for joining (it is the leading reason), then how different are the people that joined to those that didn't (probably not much) and therefore how relevant it may be as a reason.

The study highlights the usual suspects and some interesting points:
- economic incentives as positions within Al-Shabab are remunerated (crime does pay), and many of them would see it as a career path.
- propaganda, specially the perception that Islam is under threat mixed with nationalism (Christians invading the country)
- Status/empowerment, specially for marginalized youth and minority clans. Al-Shabab represents respect. It is also able to provide a sense of belonging and a common purpose (that affects retention of recruits, although the money did help)
- Fear and anger as driving emotions for recruitment 
- Peer networks are the source of contact and not family networks. Many would join through a friend and rarely through a family member. Interesting point in a society where family/clan are so important, and maybe the breakdown of that relationship is a leading driver
- There remains a high trust on elders, even though much of Al-Shabab rhetoric and actions tend to undermine that structure. On the other hand they see government as self-interested and illegitimate, being revolt a necessary step.
- Regret (or lack of). Many of the former fighters regretted being caught, and not necessarily joining Al-Shabab (probably linked to the sense of belonging that the organization is able to impress)
- Intelligence. The ability of Al-Shabab to collect intelligence (and act upon it, orchestrating complex operations) has been recognized for a while. That ability is not only an operational asset but it probably also reinforces their capacity to identify and recruit individuals.

The recommendations of the study are pretty much standard (and oft repeated), and yet again barely followed.