How Much Can We Generalize from Impact Evaluations? - Working Paper

For those of us logarithmically challenged, Vivalt's "How Much Can We Generalize from Impact Evaluations?" is a daunting task. The topic is quite relevant, the use of impact evaluations (and specifically Randomized Control Trials as the 'golden standard') have been growing in the evaluation of development interventions. However, a common counter-argument has been on the ability to generalize the results to other contexts. This paper does try to address that question, unfortunately (or maybe expectedly given the trend of mathematical jargon in social sciences) the paper is buried under equations, tables, regressions and the like. Meaning that a topic of great importance for development practitioners gets lost in the medium. I wouldn't be surprised if many of my colleagues give up just with the abstract: 

"Impact evaluations aim to predict the future, but they are rooted in particular contexts and results may not generalize across settings. I founded an organization to systematically collect and synthesize impact evaluations results on a wide variety of interventions in development. These data allow me to answer this and other questions across a wide variety of interventions. I examine whether results predict each other and whether variance in results can be explained by program characteristics, such as who is implementing them, where they are being implemented, the scale of the program, and what methods are used.  I find that when regressing an estimate on the hierarchical Bayesian meta-analysis result formed from all other studies on the same intervention-outcome combination, the result is significant with a coefficient  of 0.6-0.7, though the R-squared is very low.  The program implementer is the main source of heterogeneity in results, with government-implemented programs faring worse than and being poorly predicted by the smaller studies typically implemented by academic/NGO research teams, even controlling for sample size.  I then turn to examine specification searching and publication bias, issues which could affect generalizability and are also important for research credibility.  I demonstrate that these biases are quite small; nevertheless, to address them, I discuss a mathematical correction that could be applied before showing that randomized control trials (RCTs) are less prone to this type of bias and exploiting them as a robustness check." 

Those brave (or foolish) enough to have gone on would probably take away a couple of points:

- Academic/NGO scalability into government programmes is not a given. Taking into account that many of our programmes, specially those with governance components, make claims of eventual government take over/transfer (a favorite explanation for the sustainability section of proposals), an even more nuanced view is needed for future impact. Alternatively, further study is needed to find out how consistency of impact can be ensured.

- Heterogeneity. A fancy word for 'results all over the place'. Not only because of programme implementer as see earlier, but even within studies there is a high degree of variation. I believe this points out to the fact that we may not know what the underlying factors that affect the intervention are (making the control of variables much harder). Also the general underpowering (statistical) of studies, sample sizes may not be large enough to smooth out results (this is a hunch, as I am in no position to prove it or debate it), and, of course, context.

- Impact evaluations do have some predictive power in development. We would be in trouble if the result would have been negative! 

¿Qué hay de nuevo Yemen?

El primer ministro Khalid Bahah anuncio el sábado la composición (parcial) del gabinete. De 34 puestos, nueve irían al General People’s Congress (GPC), nueve al Joint Meeting Parties (JMP), seis al Movimiento del Sur y seis para los Houthi. Los ministerios de asuntos exteriores, defensa, interior y finanzas serán asignados directamente por el presidente Hadi. El JMP ha protestado por la distribución de ministerios y no está claro hasta qué punto colaboraran en la formación de gobierno.
 
Los incidentes violentos (incluyendo ataques suicidas) continúan en varios lugares. Milicia Houthi capturo el castillo de Al-Hodeida tras combates con Hirak Al-Tuhami (movimiento basado en zona oeste del país). Houthi y Al-Qaeda/aliados tribales se han enfrentado en Ibb y Al Bayda (alrededor de la ciudad de Rada)

Protestas y manifestaciones pidiendo la independencia continúan en las provincias del sur. El Movimiento del Sur ha fusionado sus consejos en uno, el Consejo Supremo del Revolucionario y Pacifico Movimiento del Sur.

Non-Farm Enterprises in Rural Africa: New Empirical Evidence - Working Paper

Probably the biggest takeaway from Nagler and Naude's "Non-Farm Enterprises in Rural Africa: New Empirical Evidence" is the recognition that 'Africa is not a country', or as they put it: Heterogeneity. They compare data from six countries and data diverges significantly as dynamics are tied to local context, not only at national level but also at sub-national. Nevertheless, the issue of non-farm enterprises remains key in many African (and other) countries, and this paper highlights it quite well.

 

Abstract:     

"Although non-farm enterprises are ubiquitous in rural Sub-Saharan Africa, little is yet known about them. The motivation for households to operate enterprises, how productive they are, and why they exit the market are neglected questions. Drawing on the Living Standards Measurement Study -- Integrated Surveys on Agriculture and using discrete choice, selection model and panel data estimators, this paper provide answers using data from Ethiopia, Niger, Nigeria, Malawi, Tanzania, and Uganda. The necessity to cope following shocks, seasonality in agriculture, and household size can push rural households into operating a non-farm enterprise. Households are also pulled into entrepreneurship to exploit opportunities. Access to credit and markets, household wealth, and the education and age of the household head are positively associated with the likelihood of operating an enterprise. The characteristics are also associated with the type of business activity a household operates. Rural and female-headed enterprises and enterprises with young enterprise owners are less productive than urban and male-owned enterprises and enterprises with older owners. Shocks have a negative association with enterprise operation and productivity and a large share of rural enterprises does not operate continuously over a year. Enterprises cease operations because of low profits, a lack of finance, or the effects of idiosyncratic shocks. Overall the findings are indicative that rural enterprises are "small businesses in a big continent" where large distances, rural isolation, low population density, and farming risks limit productivity and growth."

Some points to note from the study:

- entrepreneurship have had a very urban focus, and when it comes to the rural areas the focus is on agriculture. The study show the importance of non-farm enterprises in smothing shocks and providing additional income

- Most livelihoods and entrepreneurship interventions focus on the individual. However,
in this context, enterprises are a household activity (both in decision making and operation), specially taking into account distribution of labor according to seasonality (planting and harvest), number of available workers and household tasks (and gender-based allocations).

- Response to shocks (drought, price, conflict) is a major reason for enterprise creation, however not all shocks have the same effect on the choice of sector. Many differences are country specific (different responses to the same kind of shock). Moreover, productivity of enterprises born out of necessity (shock response) is lower than those out of choice. This may help us rethink the programming 'mix' regarding both prevention and response to crisis.

- In some cases, education was positively correlated with entrepreneurship, probably due to the fact that educated individual do not have a comparative advantage in a rural setting where waged employment is limited and therefore reallocate into self-employment. The role of education (or the kind of) in rural settings have to be thought further.

- Each non-farm sector has its own profile. The variables studied (education, number of adults in the household, age and credit access) have different effect depending on the sector: agribusiness, sales, trade, restaurant, professional services, etc...

- Distance to population centers. Highlighting the importance of rural secondary towns and cities, as enterprises located up to 10 km from a population center are the most productive (on average urban enterprises are more productive than rural). Productivity drops significantly when they are more than 50 km away. Large gains in poverty reduction have actually been linked to secondary cities rather than major cities or capitals.

-Youth. Younger households are not only less productive but also less probable to engage in non-farm entrepreneurship in comparison with middle age groups. To note this is talking about household heads, and not necessarily individual engagement in activity.

- Seasonality and enterprise life. Agriculture remains the key driver in many rural settings and non-farm enterprises adapt to that cycle. Enterprises, in many cases, appear/grow and disappear /decrease according to the planting and harvesting needs, and not necessarily seen as permanent activities

- Exit reasons. The most self-reported reasons of enterprise termination are low profitability, lack of finance, unreliable supplies, as well as the impact of  death or illness in
the family (specially in settings with limited insurance or social protection)

While this paper provides and overview might be good for getting ideas where to look further, we would be better off going into country specific studies when it comes to programming and policy recommendations/development.

Interesting links

Latest round of interesting articles:

- Why are you armed? Gun culture in Yemen

- Sun Tzu, entrepreneurship and the Austrian School (economics) in "The Political Economy of 'The Art of War'"

- Genetic testing in Rapa Nui (Easter Island): Polynesian, Native American and European contacts.

Do Traffic Tickets Reduce Motor Vehicle Accidents? - Working Paper

Traffic tickets are very unpopular among drivers, but to what extent do they affect behavior? do they reduce fatalities? These questions are addressed by Luca's paper "Do Traffic Tickets Reduce Motor Vehicle Accidents? Evidence from a Natural Experiment".

 
Abstract:     

"This paper analyzes the effect of traffic tickets on motor vehicle accidents. OLS estimates may be upward-biased because police officers tend to focus on areas and periods with heavy traffic and thus higher rates of accidents. This paper exploits the dramatic increase in tickets during the Click-it-or-Ticket campaign to identify the causal impact of tickets on accidents using data from Massachusetts. I find that tickets significantly reduce accidents and non-fatal injuries. However, there is limited evidence that tickets lead to fewer fatalities. I provide suggestive evidence that tickets have a larger impact at night and on female drivers."

 

The deterrence effect of law enforcement is a much debated issue, and traffic tickets are even more controversial on their own as they are/could be used as revenue source. Nevertheless, the stated purpose of the fines is to enforce traffic regulation and, if we are to believe the ad campaigns, prevent fatalities. Using a seat-belt law and enforcement campaign is a nice natural experiment, as seat-belt are designed for preservation of life.

The study shows that tickets do lead to a reduce number of accidents but not necessarily to a decrease in fatalities. This may be due to the allocation of police resources, that may be placed in locations and times that may not prevent fatal accidents (i.e. nearly 50% of fatalities occur at night).

¿Qué hay de nuevo Yemen?

El Comité encargado de preparar el borrador de la nueva Constitución de Yemen dice haber completado el 80%, desafortunadamente la parte restante también es la más espinosa con temas como federalismo o inmunidad de altos cargos. Se espera un primer borrador para noviembre, y el comité se encuentra en Abu Dabi.

El Consejo de Seguridad de Naciones Unidas podría imponer sanciones a prominentes yemenís acusados de fomentar la inestabilidad. Aunque la amenaza se ha hecho múltiples veces en el pasado, eventos recientes no hace sino aumentar las posibilidades. Entre la lista de sancionables esta el ex presidente Saleh (acusado de colaborar con los Houthi), su hijo Ahmed Ali (general a cargo de la Guardia Republicana antes de ser nombrado embajador en EAU), el líder Houthi Abdulmalik al-Houthi, su hermano Abdulkhaleq al-Houthi y el líder militar Houthi Abu Ali al-Hakem.

El movimiento Hirak ha establecido campamentos de protesta pidiendo la secesión tanto en Aden como en Mukalla (Hadramout). El movimiento independista está cogiendo fuerza, especialmente en reacción al avance Houthi, y han pedido al gobierno que retire a las tropas y administración del Sur para finales de mes... algo improbable, ya que significaría la desaparición del gobierno a efectos prácticos. El movimiento Tehama (costa occidental de Yemen) esta organizando manifestaciones pidiendo la retirada Houthi de Al-Hodeyda y la región.

La expansión Houthi empieza a encontrar más resistencia. Tribus locales en Ibb están expulsando o capturando militantes Houthi, y, en otras partes de la provincia, Al Qaeda ha tomado múltiples poblaciones tras combates con los Houthi. No se sabe hasta que punto hay colaboración entre la tribus locales y Al Qaeda, pero no es descartable que sean aliados de conveniencia. Los combates continúan en Al Bayda, principalmente alrededor de la ciudad de Rada.

El oleoducto en Marib fue saboteado por militantes tribales, provocando un paro temporal en el bombeo.

Interesting links

Latest round of interesting articles:

- How to talk to terrorist or rather why we should.

- The evidence on travel bans for diseases like Ebola is clear: they don't work

-Taking inspiration from the Torah, "Megillath Esther and the Rule of Law: Disobedience and Obligation"

-A Development Agenda without Developing Countries? The Politics of Penurious Poverty Lines, by Lant Pritchett