This is a critique of a qualitative, protocol-analysis empirical study by Sarasvathy, Simon, and Lave (1998) that I wrote on 2017-09-22 for the class, EDF 7475: Qualitative Research in Education taught by David Boote, Ph.D. at University of Central Florida.

EDF 7475 Article Critique One
Richard Thripp
University of Central Florida

Article Citation

Sarasvathy, D., Simon, H. A., & Lave, L. (1998). Perceiving and managing business risks: Differences between entrepreneurs and bankers. Journal of Economic Behavior & Organization, 33, 207–225. https://doi.org/10.1016/S0167-2681(97)00092-9

Summary

The authors wrote a series of business problems, some involving risk of financial loss, and others involving uncertainty of workers dying or getting cancer. They administered these written problems to successful entrepreneurs (n = 4) and seasoned bankers (n = 4) who were recruited from continuing education participants and alumni of Carnegie Mellon University. The authors used verbal think-aloud protocols to analyze participants’ responses and thought processes to the written problems. Through cluster and protocol analyses, the authors concluded that entrepreneurs and bankers conceptualize risk and uncertainty differently. Bankers tend to hold returns fixed and try to decrease risk, while entrepreneurs accept the given levels of risk and focus on increasing returns. When confronting potential cancer or death due to carcinogens and other hazards in the workplace, bankers spoke in the third person and restricted their problem space to financial, legal, and ethical issues, often refusing to make a decision; contrastingly, entrepreneurs put their personal values first and looked for external solutions (e.g., being acquired by a larger company) to provide the millions of dollars required to improve safety.

Contribution to the Field

Methodologically, this article contributed by using a qualitative approach, which is unusual in finance and economics. The stark conceptual differences that emerged from protocol analysis of the bankers as compared to the entrepreneurs contributes to future studies of different financial perspectives, and more broadly, to management and leadership studies. Moreover, the business problems the authors developed are of value, although unfortunately from a web search it appears no one else has ever used these problems in the 21 years since this article’s acceptance.

Strengths and Weaknesses

Ericsson (2002) says a protocol analysis should ideally include other indicators like response times and brain activity. While the authors verbal coding and analysis was parsimonious yet sufficiently detailed, they did not include any information on how long respondents took on each question nor the entire instrument. One strength was that many utterances along each cluster dimension were included in Table 1, although the amount of time respondents spent pondering the problems could also have been included. However, the inclusion of quantitative participant-level statistics in Tables 3–4 is laudable and should be followed by other qualitative researchers, albeit the value here is dubious due to the small amount of data.

In delivering the instrument, procedural rigor was lacking. This is abundantly clear in Appendix A, where Banker 2 mistook an amount in Problem 5 to be $1 million instead of $5 million, because “apparently he was given the copy of the problems used by the previous subject who, after completing the protocol, had discussed the possibility of making it one million and had changed the number 5 to number 1” (pp. 224–225). Another oversight is that Appendix B: Results of Cluster Analysis only refers readers to statistics in Tables 3–4 which were included in Appendix A, but these tables should have been moved to Appendix B. Finally, participants may be identifiable—we know the entrepreneurs founded companies with $5–30 million annual revenues—but social desirability bias is never mentioned, despite the nature of the problems. Given the prestige of the journal and institution, these shortcomings are surprising.

There is no getting around the fact this was a convenience sample. Entrepreneurs who were participating in the authors’ continuing education program were solicited to participate, and bankers were alumni “selected based on geographical accessibility and convenience of scheduling” (p. 208). Moreover, the design of the study suggests the authors may have held a priori assumptions about the differences between bankers and entrepreneurs. Further, the writing and design of the problems, including rationale for offering only di- or trichotomous choices, is absent. It is not even clear the authors wrote the problems— “five problems were used” (p. 208) is what they state, although I surmise they wrote the problems based on an absence of web results when searching portions of the exact text of the problems, besides this article itself.

Nevertheless, this was a concise, enjoyable read that resulted in surprising findings and insights. For example, in Problem 1, I find bankers’ preference for Project 2 stunning given the high probability of low returns contrasted with only a 0.5–0.75% increase in summed returns, based on cumulative probability. Even with discretionary monies, accepting enormous uncertainty for such marginal gains seems overly rational—almost inhuman. Finally, it was smart of the authors to note that causality could flow counterintuitively—perhaps bankers self-select due to their pre-existing aversion to risk, rather than developing the aversion on the job?

Contribution to My Understanding

This article showed me that conducting a verbal protocol analysis is not an insurmountable challenge. The authors’ coding scheme was concise and readily accessible, while their consistent emphasis on participants’ quotes gave me a clear window into several perspectives and lines of reasoning. Moreover, the article made it starkly clear that qualitative research involves many value judgments—there were certainly hundreds of utterances the authors did not see fit to include, and a detailed copy of their encoding protocols is only available upon request. However, trusting in their judgments and techniques, I feel reading their selections and analyses is preferable to having the raw recordings or transcripts. Finally, I now understand that protocol analysis can be used not just for comparing experts and novices, but also different kinds of experts, and that many inferences can be inductively drawn from the results.