Qualitative Methods in Economics: "You Can Observe a Lot Just by Watching"

By Cahal Moran

If there’s one method economists have neglected the most, it’s qualitative research. Whereas economists favour mathematical models and statistics, qualitative research seeks to understand the world through intensive investigation of particular circumstances, which usually entails interviewing people directly about their experiences. While this may sound simple to quantitative types the style, purpose, context, and interpretation of an interview can vary widely. Because of this variety, I have written a longer post than usual on this topic rather than doing it a disservice. Having said that, examples of qualitative research in economics are sadly scant enough that it doesn’t warrant multiple posts. In this post I will introduce qualitative research in general with nods to several applications including the study of firm behaviour, race, Austrian economics, and health economics. More than usual I will utilise block quotes, which I feel is in the spirit of the topic.

Qualitative vs Quantitative: What’s the Difference?

Thanks to Raul Pacheo-Vega I found a seminal paper A Tale of Two Cultures: Contrasting Quantitative and Qualitative Research by Mahoney and Goertz, and it really is worth your time. Coming from political science, which more than any other discipline has experienced a schism between the two traditions, the authors sketch out how each tradition approaches 10 different areas of research. They seek to understand them as distinct approaches with their own merits and drawbacks rather than to elevate one or the other. They also point out that the names are misnomers: qualitative research may make use of numbers, while models and stats do not speak for themselves and many words are spent on their interpretation. The authors suggest some alternative labels to clarify things.

The taxonomy that most cemented my own understanding of this debate was that of probabilistic versus case-based reasoning. While probabilistic approaches seek to understand general populations by making inferences from representative samples, case-based approaches seek to understand a specific situation through in-depth study. As the authors note, there is no reason to think one approach is more ‘scientific’ than the other: the question of what caused the space shuttle Challenger to explode – a particular case - was considered scientific by the Nobel Prize winning physicist Richard Feynmann. While quantitative researchers would raise doubts about the generalisability of the small samples typical in qualitative research, qualitative researchers would emphasise that causal pathways are often specific to small populations, and general statements about causality are difficult to make with the number of observations typical of a quantitative study.

In order to discern causality, case-based reasoning utilises logic and set theory, thinking about cause and effect in terms of necessary and sufficient conditions. The goal is usually to understand every possible path to a given outcome, sometimes deemed the ‘causes of effects’. One new idea for me was what are known as INUS conditions – Insufficient but Nonredundant parts of an Unnecessary but Sufficient condition. This sounds confusing, but is easily illustrated with the example of a housefire. There may be more than one way a house fire can start and we’d ideally like to know all possibilities if we’re to prevent them. A house could burn down through a combination of short-circuiting electrics causing its wooden frame to catch alight; or through a combination of a furnace causing a nearby can of lighter fluid to explode. The short-circuiting, wooden frame, furnace, and lighter fluid are each an INUS condition which must be combined with a specific other INUS condition to create a fire - yet the fire could still occur without any of them, through a different pathway.

Understanding this would be virtually impossible with probabilistic reasoning because the idea of each variable’s quantitative impact on the fire – the ‘effects of causes’ - makes less sense than the idea of which variables are necessary or sufficient to cause the fire. The short-circuit does not have a quantitative impact on the ‘probability of a fire’; its impact depends on the other conditions. In a regression model the number of combinations of variables which can produce a given change in the outcome is practically infinite, but the set-theoretical approach is more restrictive.

There are countless economic questions where the case-based approach should reign, particularly historical questions such as ‘what causes countries to industrialise?’, where quantitative methods are famously poor. In case-based reasoning a single example can disprove a causal story: if the combination of conditions a, b, and c are thought to necessarily cause industrialisation, a single country in which a, b, and c are present but industrialisation has not occurred will suffice to disprove the theory. In contrast, such a country could be subsumed as ‘measurement error’ in a quantitative study because they deal with averages and distributions rather than logical necessities.

Considering the different conditions and possibilities – whether in an open-ended interview, case study, or ethnography – almost invariably entails speaking to real people (something economists might be reluctant to do). Creating an atmosphere of constructive collaboration is crucial and in many ways the conversation is itself the research: a ‘learning experience’ for both parties, as Tilda Gaskell puts it. An interviewer who is too rigid in their questioning is unlikely to create a rapport and elicit creative or interesting responses; on the other hand, if interviews are allowed to deviate too much from the plan they can be unsystematic and difficult to interpret. It is also important to note that issues of power, race, and gender can affect the outcome of an interview greatly (another set of considerations economists may struggle addressing).

Qualitative Research as a Corrective

In general, qualitative methods are better placed to study groups who have not traditionally been featured in economics – women, people of colour, and workers in the informal economy (such as sex workers). Because quantitative data is costly and difficult to gather, what has been gathered is a reflection of historical priorities, which almost by definition haven’t favoured marginalised groups. To this end, Trevon D. Logan has written a paper which combines qualitative and quantitative methods to study cotton picking in the Jim Crow South of the USA. He writes:

“Unfortunately, quantitative data is not a solution to this problem. The high empirical standards of contemporary economic history could be inappropriately applied to the subject of race. The implicit claims of the objectivity of quantitative data do not always hold up against the racial stain of American history. The reliance on quantitative data also limits the questions we ask about the relationship between America’s racial and economic history. I believe that qualitative empirical evidence is the missing element of African American economic history. Rather than supplementary, qualitative evidence should be a primary source of empirical evidence in African American economic history. Doing so may resolve several persistent problems in the field—the omissions and incompleteness of quantitative data, the lack of accuracy in the data we do have, and the nagging question of agency in African American history generally.”

Logan uses data from his own family records and interviews with family members to estimate cotton-picking rates and finds that their productivity was quite similar to modern estimates of the productivity of slaves. But his qualitative data tells him much more: the practice of cotton-picking was a marker of being backward and had long-lasting psychological consequences. The constant pressure to pick in order to survive meant children were forced to work - and faced with corporal punishment should they not work, or not work hard enough. This degree of marginalisation and terror experienced might make us question the sharp line drawn between slavery and emancipation, at least for some aspects of life. Logan has also written a book on male sex work which combines quantitative and qualitative research, which I have started reading and cannot recommend highly enough.

Emily Chamlee-Wright has also used qualitative methods to study race-related issues, including the aftermath of Hurricane Katrina and the informal economy in Sub-Saharan Africa. She likens qualitative research to the Austrian economists’ notion of entrepreneurial discovery: in both cases, “the relevant knowledge is never given but instead must be discovered through intellectual trial and error.” If the market is characterised by complexity, uncertainty, and discovery, why would knowledge about the market be any different? In the instance of Katrina, Chamlee-Wright notes that the researchers went in expecting the challenge of physical rebuilding to be highest on the list of surviviors’ concerns, but actually what they found was that the planning procedures for how and when to rebuild were the most frequently cited issue. She has expanded elsewhere on the relationship between qualitative methods and praxeology, a favoured method of the Austrian school which emphasises meaning and purpose. But a full elaboration of that point is perhaps best saved for another time.

It is worth pausing to reflect on the dismissive attitude most of the economics profession has towards qualitative work. It is virtually inconceivable that an economist could write their thesis using qualitative methods; there are very rarely any modules devoted to it on degrees; and it is extremely rare to see qualitative research in journals. More than this, such methods are routinely dismissed as “not rigorous” or “unscientific” in conversations and at conferences. As a quantitative economist this attitude infuriates me and clearly I am not the only one; Günseli Berik has also written with justified anger about such “arrogance”. She notes in the journal Feminist Economics that quantitative metrics have routinely ignored work done by women in the household and also treat discriminatory practices within firms as a ‘black box’. Qualitative approaches can help to shed light on both of these, something I noticed in my previous posts on feminist economics. It’s perhaps instructive that when researching all the qualitative research for this blog post, I noticed an unusually high number of female authors.

David Levine offered the following rebuke to the conceits of quantitative researchers:

"Regressions also have serious problems of generalizability (they predict poorly out-of-sample), subjectivity (researchers may stop specification searches when their favorite t statistic rises over 2), and measurement error (critical concepts like 'income' and 'capital' are very poorly measured). This is why research is hard-and why we should believe only findings obtained with multiple methodologies.”

Martha Starr agrees that a mixed-methods approach is the most desirable. For example, qualitative research can pave way for the gathering of quantitative data by helping researchers figure out what’s important. But it can do more: it can help us to understand how things work (or don’t) by looking beyond the figures.

"You Can Observe a Lot Just by Watching"

Economics has traditionally been about agents who have both objectives they want to pursue and constraints that limit how well they can pursue them. Susan Helper points out that asking people directly is a pretty good way of finding out their objectives and constraints, using a law firm as a case study. Puzzled by the fact that many workers complained about long hours but no firms offered them, researchers asked managers about it. From these conversations they realised that working long hours sends a signal to your superiors that you were a hard worker, which can result in everybody working an inefficiently high number of hours. Helper discusses a number of other helpful (!) examples, including one from the economist Claudia Goldin when she visited an auto-parts manufacturer:

"I didn't have any particular expectations going into the plant, but I remember vividly looking down from a mezzanine from which you could see the whole shop floor. As I looked down, I realized I was observing-in one moment-the transition from 19th century technology to 20th century technology. I could see the relative increase in the demand for skill just scanning across the room. The 'continuous process' machinery required lots of skilled labor to set up the machines and mechanics to maintain them; there were few operators....The old-fashioned areas [making similar parts], however, were filled with semi-skilled workers and almost no skilled workers. The scene sparked my imagination and I wrote two papers (Goldin and Lawrence Katz, 1996, 1998). For years I had been reading the history of technology, but it wasn't until I went to Pollak that I made the connection that adoption of continuous-process technology was complementary to skill."

Two well-known economists, Alan Blinder and Truman Bewley, wrote probably the most famous qualitative studies in economics with their respective books Asking About Prices and Why Wages Don’t Fall During a Recession. Both had a similar goal – to ask firms exactly why their prices or wages didn’t fluctuate as much as economic theory predicted – and both revolutionised our understanding by introducing notions such as reciprocity, fairness, and trust into a field which had traditionally side-lined them. Bewley has since detailed and defended the use of interviews as an empirical tool in economics in a nice paper, which notes (among other things) that economic actors often found economic theories hard to understand or silly, which is itself quite revealing. Ronald Coase is another prominent economist whose canonical theory of the firm was developed after a series of on-site visits.

Despite the relative success of these examples the approach remains rare and arguably the insights gleaned haven’t been fully appreciated in the theory of the firm - something I argued in my earlier post on firm behaviour, which drew from qualitative research. One thing which becomes clear from visiting firms and speaking to economic actors is how much variety there is in how people behave and how organisations operate, and a good example of this is healthcare.  Health system vary so much between and even within countries that qualitative research may well be necessary to understand it. Nevertheless, a paper in the journal Health Economics found that the percentage of papers using qualitative methods in the field was only in the single digits, and called for more qualitative research in the field:

“Health economists frequently argue that the well-known difficulties of uncertainty and incomplete information that characterize the health care sector have given rise to the development of a highly complex and diverse set of institutions that differ markedly across time and space. In general, researchers will wish to generalize their findings and to draw lessons for future policies indicative of the effects we can expect for other populations in other institutional settings. Such arguments derive their strength from knowledge of the context and processes, which generated the observed effects, and it is here that qualitative research may plausibly help in generating this knowledge.”

In her Master’s dissertation Lindsay Larsen used qualitative interviews of smokers regarding tobacco tax increases to test Gary Becker’s theory of rational addiction. Although many, including King, regard the idea that addicts rationally plan their addiction as absurd, the interviews revealed some congruence between the theory and between peoples’ perceptions. People were aware that their consumption now would likely increase their addiction in the future, and many seemed to respond to future tax rises as the theory predicted. On the other hand, addicts expressed internal struggle and sometimes straight up inconsistencies in their reasoning, which it’s fair to say are not features of the rational theory. It’s pretty hard to cling to any theory in its entirety when you actually go out and run it by real people.

Michael J. Piore has argued that by interviewing economic actors directly, qualitative research can help us to investigate key assumptions of economic theory like this. I mention Piore because he also entertains the following more nihilistic possibility, which is a thought I’ve had myself:

“In interpreting interviews, I do not think sufficient attention is ever given to the possibility that the world is really chaotic; it doesn’t fit anybody’s models, not those of the social scientist and not those available to the actors.”

At the very least, qualitative methods are an antidote to those social scientists who presume they have the world figured out.