Case Study Methodology

3I - Case Study Methodology

Essex Summer School Exam 2011

1. Assessing research design
Lily Tsai, “Solidarity Groups, Informal Accountability, and Local Public Goods Provision in Rural China” (American Science Review, May 2007).

a) To what extent is the empirical research guided by theory?

A good case study promotes an active dialogue between theory and data. In her case study of rural China, Lily Tsai manages to keep this dialogue alive. Tsai’s aim is clearly stated: to investigate how public goods can be provided even in non-democratic systems where the public’s influence over local government is mostly limited to informal mechanisms. Specifically, she looks at the incentives of local public officials to provide public goods in such contexts. However, it can be questioned to what extent the theory was outlined to fit the data. Scholars argue that theoretical propositions should be a guide to empirical exploration (Hay, 2002; Yin, 2009). In that sense, as we will see below, we may ask to what extent Tsai’s study rather is an example of the opposite, namely that the empirical research seems to partly guide the theory.

Tsai proposes a model of what she calls “informal governmental accountability”, according to which local officials may have strong incentives to provide public goods, even when formal government accountability is weak, if citizen solidarity groups award them moral standing for doing so. Her hypothesis is hence that villages with greater social solidarity will experience better governance due to a higher level of social trust and mutual obligation. To create the needed conditions and incentives for officials to provide public goods, solidarity groups must be “encompassing” and “embedding”, that is they need to be open to everyone in the community and incorporate the local officials as a group member.

In all case studies, it is essential to ask and answer the question “what is my case, a case of?” As implied by her research question (“why would government officials in authoritarian and transitional systems, where formal democratic and bureaucratic institutions of accountability are often weak, ever provide more than the minimum level of public goods needed to maintain social stability?”), Tsai’s study is a case of how members of society may organize to create incentives to obtain public goods provision despite the lack of formal access systems and democratic control mechanisms. However, this crucial aspect of weak democratic institutions, as addressed in her question, seems to have been forgotten halfway through Tsai’s outline of her hypothesis. When she later on looks at the data analysis, makes inferences based on the results and tests the theory, it seems that the accountability aspect of her research question has been forgotten. When Tsai for example states that “[t]his paper simply suggests that all things being equal, solidarity groups with these two structural characteristics [encompassing and embedding; my note] have a positive impact on local governmental public goods provision” (p. 361), it seems that ‘what the case is a case of’ has shifted in due course of her paper and the issue of a non-democratic context has been dropped. It hence becomes difficult for the empirical research to be guided by theory when the framework of the theory is shifting. We can partly find the origin of this flaw among the background assumptions for Tsai’s hypothesis.

Case study methodology implies a strong combination of intense theorization and intense observations. These are both present in Tsai’s research, as it is generally well-designed and minutely implemented through an impressive use of mixed method where qualitative and quantitative data build on each other. However, some assumptions that form a crucial part of the basis for Tsai’s theory are not solid enough.

Geddes (2003) argues that “[a]t the stage of theory development, it is virtually impossible to avoid ‘overfitting’, that is, tailoring arguments to fit the circumstances found in particular cases.” In her great enthusiasm for a beautifully formulated hypothesis, Tsai seems to be walking straight into this trap. In fact, for the purpose of outlining her very concise and well-formulated hypothesis, Tsai takes some great leaps of faith in her theoretical discourse and makes assumptions that are not solidly based in theory. Especially, Tsai claims that “[i]n localities with encompassing and embedding solidarity groups, citizens and officials are more likely to share a common set of ethical standards and moral obligations,” yet without showing in what way this may be true for the context of her selected case.

Tsai makes reference to various studies done in regard to among others solidarity groups, types of prestige and moral standing, collective action problems, and developmental states, and she gives convincing hypothetical examples to explain her point. Yet, in regard to the most interesting and crucial aspect of her case – namely the successful provision of public goods in a non-democracy – she does not make cross-references to other cases or theories in the presentation of her hypothesis. Since the quoted inference is supposed to build a theoretical foundation for a case study in an undemocratic regime, where the lack of a free civil society may create different conditions that in relatively freer societies, this becomes a vague assumption. Unfortunately, this undermines the results of the study, as it is a crucial basis for building the theory of the research study. Tsai does make reference to so called “village democratic reforms”, but does only briefly explain what this means (p.363). When she later in the study draws conclusions in regard to broader aspects of theories of democracy (p.370), one can question its validity as we have not satisfactorily been told how democracy in this case has been defined. Neither is it mentioned how solidarity groups in the villages that are subject to so called “democratic reforms” are affected by the generally oppressive milieu of an authoritarian regime like China. To make any inferences about how democratic institutions work or do not work in regard to civil action does hence not seem appropriate in a study where this aspect has not been firmly integrated into the hypothesis or in the theory-testing. Rather, the democracy/accountability argument seems to be “pulled out” in an inconsistent way when it serves the purpose of confirming parts of the theory, when the data allows it.

According to Geddes, theories should be tested on a few other cases in the initial hypothesis development to try to exclude so called “error terms”, meaning factors that do not have a causal influence on the selected case. This may be considered a too rigid criterion, as it may many times be impossible to execute. In the case proposed by Tsai, however, it would have increased the credibility of her hypothesis if she would have included a short discussion of the applicability of her theory on other “least likely cases”, meaning other undemocratic regimes and contexts where civil society is closely controlled by the state.

Tsai’s research design is otherwise solid. It would hence be reasonable to ask if a discussion regarding communalities of ethical standards and moral obligations in other authoritarian contexts was omitted, not only because of the difficulty of finding similar cases, but also because of the problems it might have created for her theory assumptions in an otherwise solid theoretical framework. Hence, at the cost of what seems to be the temptation of a neatly formulated hypothesis, Tsai puts the construct validity and reliability of her study at risk. Tsai puts the cart before the horse and partly lets the empirical research guide the theory of her study.

b) What are the pros and cons of the case selection?

Most scholars agree that a research design should contain a minimum of four components: research question, theory, data and data analysis (King, Keohane & Verba, 1994). George & Bennett (2005) also underline the importance of case selection as a task of its own. As Gerring (2007) elegantly puts it, “[c]ase study analysis focuses on a small number of cases that are expected to provide insight into a causal relationship across a larger population of cases. This presents the researcher with a formidable problem of case selection.” In other words, case selection is no simple matter. In addition, as Geddes (2003) has demonstrated in further detail, the case selection will unavoidably affect the results of a study. Hence, not only is it a difficult task, but case selection is also an important step to ensure solid research. I start here by underlining the challenge that Tsai, like all researchers, is up against. Nonetheless, the case selection of Tsai’s study, or rather its methodological justification, is the weakest link of her research project.

Tsai’s case selection can be divided into a primary case, the country of China, and sub-cases made up of the in the villages (and their counties and provinces) where the data collection was done.

In regard to the primary case selection of China, Tsai gives little methodological justification for this choice. As argued by Geddes (2003), “selecting cases without giving careful thought to the logical implications of the selection entails a serious risk of reaching false conclusions.” Since we are given no background to how the selection was made, we can only guess, based on our own (in my case limited) knowledge about the Chinese context as related to the research question. If one is generous, one could assume that the choice of China would be a crucial ‘least likely’ case (“a case predicted not to achieve a certain outcome, and yet it does so”; Gerring, 2007), in that public goods provision is not expected to be fulfilling in an undemocratic regime. Again, this is only the reader fumbling in the dark, as Tsai does not give more flesh on the bone for further understanding of her selection.

As the methodological justification of the primary case selection in this sense is missing, Tsai undoubtedly puts the conclusions of the study at risk. The introduction to the research design (p.357) certifies that “[c]ontemporary rural China provides an ideal setting to examine the factors that affect the quality of local governance because of the tremendous variation in the performance of village governments”. Here we are only provided with an explanation of why China is an ideal case for answering only a part of the research question. Neither is this truly a justification, nor does it present any form of method in how this case was selected. So, since Tsai’s research follows a theory-testing rather than a theory-building scheme, how do we know that China really is a good case for testing her theory? In regard to the primary case selection, it is hence more difficult to find pros than cons.

In regard to the sub-selection of cases, namely the four provinces, eight counties and 316 villages in rural China, Tsai presents the reader with very solid methodological justifications of her selection. She formulates both how the provinces, counties and villages were selected, and why. The provinces were chosen to “reflect differences in levels of economic development as well as regional differences” (p.357), while the two counties within each province were “selected to vary in model county status for village democratic reforms but to have similar economic and geographic characteristics” (idem). In this sense, the selection of counties can be defined as a “most different system design” (Landman, 2006). The guiding criterion for case selection should be the relevance to the research objective of the study, and to provide the kind of control and variation required for the specific problem (George and Bennett, 2005). These criteria for case selection are fulfilled in this part of the case selection by Tsai’s coherent and methodologically strong justification.

The rural villages, where the data was collected, were selected through a random stratified sampling procedure based on income per capita. To choose as many as 316 villages and hence increase the study objects, strengthens the internal validity of the study. The hypothesis is that villages with greater social solidarity will experience better governance. Hence, villages that show a high level of social solidarity, but nevertheless are least likely to develop good governance, constitute crucial least likely cases. Tsai does not categorize the case selection herself, which can be considered a con of the selection process.

Prior to the random sampling of the villages, Tsai had already made a pre-selection of the sample based on the approval of the local government of the data collection. Hence, only villages with an official “OK” could be selected. Tsai does not address the fact that collecting data exclusively from villages where the local authorities approved of her study could by itself skew the research results. Intuitively, one would think that inhabitants of villages where the local government accepts questioning by outsiders, will perceive their authorities differently that inhabitants of villages where the government would be reluctant to external researchers ‘poking around’. The lack of questioning how this village case selection could affect the research results is a con in itself.

c) What are the tools for increasing leverage used by the author?

Tsai’s study is an impressive example of how to increase leverage of a case study. Above all, Tsai uses various forms of triangulation (Tarrow, 2004), both of data and of methods. The use of “mixed method” (White, 2008) and how she combines qualitative and quantitative data to verify her results is an example of this. In regard to the statistics, Tsai gives a clear outline of the use of the dependent variables and the use of control variables (geographic and demographic; economic; democratic and bureaucratic institutions). This and the statistic estimation contribute to increase the leverage of testing the hypothesis. As Tsai states, “results from seemingly unrelated regression analysis are consistent with the hypothesis” (p.366). For pedagogical reasons, however, the statistics tables would have benefitted from a diagram to demonstrate the results – hence a point where more leverage could have been gained. Tsai also uses counterfactuals (p.365-366) to test her hypothesis, another a good technique to increase leverage (Levy, 2008).

d) What are the comparative merits of this single-country study, if any?

As noted by Landman (2006), “[s]ingle-case analysis tends to limit further its empirical generalizations […], but can be constructed in such a way as to contribute to larger theoretical and empirical problems.” Even if a case study is not comparative in method, it is important to identify comparative merits from its results to be able to generate broader theories that might also be applicable in other contexts.

The strength of a “least likely” case, like Tsai’s case study of China, is that all other plausible causal factors have been minimized and hence the verified results can be applicable to other contexts for comparison. Tsai’s research question is in this sense of interest for many fields, from political science to organizational theory. In addition, the objective of the research is unique in its scope and hence the theorization can be useful for other contexts where similar questions might arise. For example, the idea of informal government accountability spurred by civil action through citizen solidarity groups can be explored in many other contexts.

Without repeating my observations in answers a) and b), it should however be noted that Tsai’s research design is flawed to begin with, as the foundation of what her case truly “is a case of” is not solid enough, as the non-democratic aspects of the context are not fully taken into account. This will affect the results of Tsai’s research and arguably also the reliability of the comparative merits that the study has.

2. Can we really make strong inferences (generalize) from case studies?

There is not one definition of case study methodology, but in essence we can agree that it constitutes an “intensive study” (Gerring, 2007) of a single case or a small number of cases within a specific context (Yin, 2009). The purpose of case studies is usually to gain broader understanding of a larger class or typology of phenomena, using the conclusions for the in-depth case study to generate more general theories or inferences. If we define the case study method as “a research strategy of focusing intensively on individual cases to draw insights about causal relationships in a broader population of cases” (Poteete et al, 2010), there are clearly both possibilities and limitations to how far the case study method can contribute to broader generalizations and theories.

Inferences can be defined as using what we know to make sense of what we don’t know. The extent to which one can make strong inferences from case depends on how rigorous and “systematic” (King, Kehoane & Verba, 1994) the research design is, and that there is a strong coherence between theory, data analysis and conclusion. The definition of ‘rigorous’ in case study research, in the sense of strong validity, however depends on the school of thought. For positivists (see i.e. Yin, 2009), a case study methodology is only valid if it is for testing a hypothesis. On the other hand, the constructivist school of thought, based especially on Eisenhardt (1989), sees the possibility of building theory based on empirical research as a great advantage of the case study methodology.

I argue that case studies as a research methodology can be valid for generating useful generalizations within social sciences, especially when case studies have the objective to look closely at causal mechanisms and processes. Over the next few paragraphs I will show how we can make strong inferences from case studies. Of course, this argument must be considered within the fundamental limits of social science theories as tools for ‘creating order in chaos’, paraphrasing Shively (2004), rather than generating “scientific ‘laws’” (Landman, 2008) that apply in all context and situations such as natural sciences. The lack of such “laws” does however not mean that social sciences do not follow rigorous rules of methodology, and as I will argue, case studies can and do in fact contribute largely to general theories of the social sciences according to reliable methods of empirical research.

Case study methods can refer to both within-case analysis of single cases, and to comparisons among a small number of cases (Bennett, 2004). I will especially focus on single cases, as these are usually more prone to being criticized for not being able to provide “secure generalizations by comparing cases” (Lieberson in Bennett & Elamn, 2006). Critics, in fact, tend to claim that only comparative studies of many cases (large-N) can be reliable enough to make strong inferences and that the causal effects are more significant for robust results than causal mechanisms (see e.g. King, Keohane & Verba, 1994).

It is important to notice that case studies do not exclusively make use of qualitative data, but they can also be based on quantitative research. Due to the in-depth character of case study method, there is almost always a strong qualitative aspect to the data collection of case studies. However, so called “mixed method”, where both quantitative and qualitative data analysis is used in case studies, is increasingly gaining ground as a tool for increasing the leverage of case study research (see i.e. Fearon & Laitin, 2008; White, 2008; and Alston, 2005).

The need for solid case study research

The extent to which we can draw strong inferences from case studies does not depend on the case study methodology as such, but rather on how solid the research of a specific case study is. In social sciences, due to the complexity of the systems that are being studied, every case can in some ways be considered unique, and in other aspects have general characteristics that can be used for general inferences. In fact, case study findings are generally contingent. Hence, only under specified conditions can the conclusions be generalized beyond the type of the studied case, still depending on the rigor of the research design and the validity of the results.

The concept of validity can show the way to ‘true’ the results of a case study are, in various aspects. Shadish, Cook and Campbell (in Morton & Williams, 2008) define validity as “the approximate truth” of an inference, dividing the concept of validity onto four groups: statistical validity, internal validity, construct validity, and external validity. The first two types of validity are concerned with how robust the research is within the study itself. Construct validity concerns how valid the data inferences are for the chosen theory, which also reflects the internal research design. External validity, on the other hand, is whether causal inferences established in an empirical analysis can be generalized for other sets of data (Morton & Williams, 2008). This involves the possibility of replicating results across various data-sets. It is hence especially the external validity of a case study that is essential to ensure that we can make strong inferences from its specific set of data and conclusions.

A central question to answer in case study methodology, for each specific case study, is: “what is my case, a case of?” To be able to argue that a case study is solid, it is essential that this question is properly answered and hence reflected throughout the research design. Otherwise this will affect (and flaw) the results and conclusions, and show that the researcher is not certain of what he or she has the intent to observe. The framing of the case study is hence the first step to ensure its reliability, so that its results can be used for broader generalizations.

Critique and defense of cases study methodology

Case study methodology has long been considered ‘less-than-scientific’ by various scholars for not being representative enough to build new theories and make generalizations for broader contexts. Rather, case studies have been regarded as an aiding tool of other methodologies, but criticized as a methodology of its own. As stated by Abercrombie, Hill & Turner (in Flyvbjerg, 2006): “[…] a case study cannot provide reliable information about the broader class, but it may be useful in the preliminary stages of an investigation […]”.

Yet, social sciences is largely based on case studies, from the classical Indian case of Myrdal in development studies to contemporary classics as Putnam’s study of democratic institutions in Italy. As Gerring states (2007), “although much of what we know about the empirical world has been generated by case studies, […] the case study method is generally unappreciated, arguably because it is poorly understood.”

The perception that case studies are a poor tool for strong inferences, arguably because it is a limited method in regard to knowledge accumulation, is opposed by many scholars in the fields of social sciences. As stated by Jensen & Rodgers (2001), “case studies satisfy the need for conditional findings and in-depth understanding of cause and effect relationships that other methodologies find difficult to achieve.”

Another critique, that case studies generally are of “poor quality” (idem), is usually based on a quantitative argument: what makes a study qualify as good research is the number of quality criteria, rather than the relevance of those criteria. Hence, the critique becomes tautological in its refusal to look beyond only the criteria of quantity and in its incapacity to accept that quality in itself is an important issue to ensure the reliability of any research project.

Making inferences from case studies

When trying to directly observe theories or casual mechanisms, there is always a danger of measurement error, specification error and omitted variables in all research methods (Bennett, 2004). However, case study can provide us with “several sources of inference” through methods such as process tracing. This entails examining hypothesized causal and sequenced mechanisms as predicted by a theory, and hence verify if a hypothesis can or cannot be ruled out as an explanation for the chosen case.

The use of control variables is an example for why case studies can in fact be a fully reliable method to make inferences beyond the specific context of the chosen case (Morton & Williams, 2008). Also, the use of “most likely” and “least likely” cases (Eckstein, 1975) for case studies “allows inferences on the more general scope conditions of theories under investigation” (Bennett, 2004). Landman and Lijphart call this theory-confirming (or theory-infirming) method. In this sense, case study can in fact be used both to test and to create theories, as also indicated by van Evera (1997).

As part of a reliable research design, any case study is at the mercy of the capacity of the researcher to make a good case selection, especially avoiding case selection bias and verification bias. However, as pointed out by Collier & Mahoney (cited in George & Bennett, 2005), “case study researchers rarely ‘overgeneralize’ from their cases; instead, they are frequently careful in providing circumscribed ‘contingent generalizations’ that subsequent researchers should not mistakenly overgeneralize”.

As stated by Bennett and Elman (2006), “a key question for methodologists is how to draw causal inferences”. To be able to generalize from a specific research result, it is essential to understand the causality at play in the studied context. In regard to case studies, “causation can be thought of as a process involving the mechanism and capacities that lead from cause to an effect” (idem, p. 457).

The use of process tracing is a good tool to investigate this notion of “causes-of-effects”. Process tracing is in essence a method to observe processes of causal mechanisms within a single case (George & Bennett, 2005). By uncovering the steps of a hypothesized causal mechanism of a specific case, we can make inferences that apply beyond the specific case, since the process is does not simply rely on establishing causation through comparison. According to Gerring (2007), the strength of process tracing lies in that “multiple types of evidence are employed for the verification of a single inference”. This suits well for the complex world of social phenomena, where it is many times difficult to ensure that the assumption of ceteris pariubs can be made.

When process tracing provides a continuous historical explanation of a case, and each step is explained with clear reference to a theory, it is a confident method of inferences (George & Bennett, 2005). In addition, the more encompassing the process tracing of a study is in regard to the specific context, the more robust the research can be considered (for example, by including a general introduction of the social or historical context of the chosen case to show further in-depth understanding of the studied mechanisms). As noted by Bennett and Elman (2006, p. 460), “our confidence in the suggested explanation will be increased if process tracing finds evidence of observable implications that are inconsistent with alternative explanations”. In this sense, process tracing can be used as a form of theory-testing for each step of the research in specific situations where there are alternative (and contradicting) theories. Process tracing can also be employed as a supplementary tool to strengthen a standard research design (Gerring, 2007), hence as a form of triangulation of methods. Since process tracing tests theory against evidence again and again, with a chronological step-by-step method for the selected case, this is a robust tool for inferences. As noted by Gerring (2007), “although process tracing is always based on the analysis of a single case, the ramifications of that case study may be generalizable, and indeed may be quite broad in scope.”

As concluded throughout this exposé, if a case study has a systematic research design, which is implemented with rigor to ensure the validity of the results, strong inferences can indeed be made from case studies.