Do students respond in the same manner as professionals in behavioral experiments?

Bob Jensen at Trinity University 

March 1 message sent out across the networks

I am refereeing a paper for a journal published in another country. The author(s) make the claim that "students respond in the same manner as professionals" in behavioral experiments. If some of you can provide me with references for and against this assertion, it would help me and possibly the authors of the paper.
Bob Jensen

I received a number of very helpful replies.  The most helpful act was when a colleague in my own Department of Business Administration, Dr. Charlene Davis, walked into my office with photocopies of four of the leading papers in management and marketing on the issue of external validity of student surrogates in behavioral experiments.   Thank you Charlene!

There seems to be a spectrum of external validity.  The extremes are no-brainers.  Students are excellent research subjects when comparing the taste of Coke versus Pepsi.  They a terrible surrogates when asked to make complex decisions in circumstances where they have no real-world experience (e.g., making decisions that affect audit risk or brain surgeries).  The real dilemma arises in the middle 90% of the spectrum of external validity.  

Some of the most widely cited studies of this issue tend to be negative about external validity of research using students as subjects in actual or simulated real-world decision experiments.  One of the most widely cited articles is the Gordon, Slade, and Schmidtt paper noted below:

The "Science of the Sophomore" Revisited:  From Conjecture to Empiricism

MICHAEL E. GORDON
L. ALLEN SLADE
University of Tennessee, Knoxville
NEAL SCHMITT
Michigan State University

Academy of Management Review, 1986, Vol. 11. No. 1. 191-207

The controversy over using college students as subjects in applied research has been a topic of philosophical discourse and empirical investigation.  Thirty-two studies are reviewed in which students and nonstudents participated as subjects under identical conditions.  In studies reporting statistical tests of between-group differences, the preponderance of findings indicated that the experimental results differed in the two samples.  By contrast, no major differences associated with the type of subject were reported in the majority of studies which did not employ statistical procedures to compare the findings in the two samples.  Explanations for differences in the sample are offered, and serve as a basis for recommendations for future research.

To which Greenberg responded as follows:

The College Sophomore as Guinea Pig: Setting the Record Straight

JERALD GREENBERG
Ohio State University

Academy of Management Review, 1987, Vol. 12. No. 1. 157-159

As Campbell (1986) recently wrote, "Perhaps college students really are people...why their disguise fools many observers into thinking otherwise is not clear" (p.276).  Indeed, scientists need to recognize the role of college students in organizational research and to temper claims against their use by understanding their value.  Specifically, it should be realized that phenomena observed in homogeneously defined groups of subjects -- be they workers in the "real world" or college students in a laboratory -- may offer equal, limited potential for generalizability.  The use of such narrowly defined subject groups is preferable to more diffuse subject groups in approaching the theory-building goal -- and consequently, the application-development goal -- of organizational research.

To which Gordon, Slade, and Schmitt replied:

Student Guinea Pigs:  Porcine Predictors and Particularistic Phenomena

MICHAEL E. GORDON
University of Tennessee, Knoxville
L. ALLEN SLADE
University of Delaware
NEAL SCHMITT
Michigan State University

Academy of Management Review, 1987, Vol. 12. No. 1. 160-163

It appears that the standards for establishing external validity are not uniform, but rather differ in accordance with the type of research.  Standards for establishing generalizability of "universalistic" (Kruglanski, 1975) research, that is, investigations intended to affirm general behavioral laws about psychological processes, are probably more onerous than those for "particularistic" research, that is, investigations intended to affirm propositions about specific independent and dependent variables, under given circumstances.  In universalistic research, the experimental task is supposed to be devoid of social content and the results are expected to generalize to all humankind.  By contrast, the experimental task in particularistic research is replete with social content.  Further, the results are expected to generalize to occupants of more or less specific organizational roles.

But using real world (field) research is not necessarily a panacea for the problem of external validity or other criteria for evaluating research studies.  Dipboye and Flannagan state the following:

Are Findings in the Field More Generalizable Than in the Laboratory?

ROBERT L. DIPBOYE, Rice University
MICHAEL F. FLANAGAN, Florida State University

American Psychologist, February 1979, Vol. 34, No. 2, 141-150

ABSTRACT: The authors analyzed for content all the empirical articles from the 1966, 1970, and 1974 volumes of the Journal of Applied Psychology, Organizational Behavior and Human Performance, and Personnel Psychology to determine the types of organizations, subjects, and dependent measures studied.  Contrary to the common belief that field settings provide for more generalization of research findings than laboratory settings do, field research appeared as narrow as laboratory research in the actors, settings, and behaviors sampled.  Indeed, industrial-organizational psychology seems to be developing in the laboratory a psychology of the college student, and in the field, a psychology of the self-report of male, professional, technical, and managerial employees in productive-economic organizations.  The authors suggest that coordinated strategies of research in both laboratory and field settings are needed to construct an externally valid industrial and organizational psychology.

 I will now share with you a sampling of email responses to my question of whether "students respond in the same manner as professionals"   

Hi Bob

This is an issue of considerable debate in marketing. For the most part, even when students are consumers of a certain type of product, using them as part of the sample may make getting the study published much more difficult.

For business-to-business and organizational studies, using students as proxies for middle to upper level managers is not acceptable (at least in the marketing and management disciplines) and is seen as a serious compromise of external validity.

I have a couple of articles from management journals that highlight both sides of the debate. If you are interested in them, let me know.

Charlene Davis
Trinity University

Robert,

There are quite a few papers in accounting, auditing, and marketing concludes that students are not proper surrogates to professionals if the task performed in the experiment requires domain-specific experience. Some examples: Bouwman (1984) -- AOS Frederick (1991) -- AR Houghton and Hronsky (1993) -- Accounting and Finance Ho (1994) -- BRIA Stafford (1998) -- Journal of Advertising

I hope the information is helpful to your review.

chang, janie [chang_j@cob.sjsu.edu] 
San Jose State University

Robert,

The short answer is generally yes, except for extraordinary situations.

The best citation is Ed Locke of the University of Maryland(or formerly of UM), for support.

A former journal editor (Journal of Applied Psychology) said that Locke et al would fly though as support.

Charles Walton [cwalton@USA.NET] 

Bob:

I don't know the literature in this area, but some years ago, Earl Doderer ran the Briggs-Meyer test (a kind of personality test) on our faculty and a number of our engineering students. The faculty and the students scored on opposite extremes on at least some of the indicators. I'm sure that Earl has a better recollection, and perhaps some references.

If I recall correctly, one of the indicators scored the level of intuitive thinking. Probably because the faculty are all Ph.D.s with experience, we came out much more as intuitive thinkers.

J. Paul Giolma [jgiolma@trinity.edu] 

Bob,

Another good citation for students as surrogates is

Ashton, R.H. and S.S. Kramer, "Students as Surrogates in Behavioral Accounting Research: Some Evidence," Journal of Accounting Research, .pp. 1-13, 1980.

To summarize their results in defense of the use of subjects in decision-making studies, my co-authors and I make the following statement "Ashton and Kramer (1980) have shown that for tasks involving information processing and decision making, college students make decisions that approximate those made by the rest of the general population." Obviously, the Abdolmohammadi and Wright study requires some refinement to this basic statement. On the other hand, students are not good surrogates when you are measuring sentiments or opinions and the students are not representative of the population of interest (surprise surprise).

Hope this is helpful.

Stacy Kovar Kansas State University

 Bob, everything depends on what the behavior in question is. I can think of some behaviors where there would be no difference, but in others, of course there would be a difference.

In my own dissertation, I found a very distinct difference between students and professionals. (That publication can be found in, ahem, Florida State University's library... and is therefore, unquestionably, a "seminole work" in the area.) ;)

Let me know if you want the citation.

David R. Fordham [fordhadr@JMU.EDU]
James Madison University

 In an article published in JAL (1994), we used meta-analysis to investigate inconsistences among participative budgeting studies. One of the moderator variables we used was professionals versus students in laboratory experiments. We found no differences between the two in a budgetary context.

 Hossein Nouri 
Accountancy Program School of Business 
The College of New Jersey P.O.Box 7718 Ewing, NJ 08628-0718 Tel. (609)771-2176 Fax (609)637-5129 Email: hnouri@tcnj.edu 

 Bob

I hope the following thoughts are helpful.

I think the crux of the matter depends on what is meant by "behavioral experiments" i.e. what context are we talking about? BUT, any which way, I would be astonished if students behaved as professionals. I appreciate that a number of papers are published in august journals around the world which imply this very possibility. Making this possibility explicit is arguably another matter!

A general case is made by Dreyfus and Dreyfus in their famous attack on artificial intelligence. The broad argument is that novices follow rules while experts do what comes naturally - in other words experts know from experience what works.

Prior studies on judgement (which may or may not be the same as behavior depending on what you might mean by 'judgement' and 'behavior') is not encouraging for the authors. A central question is 'what is the difference between students and professionals?'. A straightforward answer, and one which follows from Dreyfus and Dreyfus' general point, is 'experience'. A substantial body of literature has shown experience to be a two edged sword but professionals ARE decidedly different.

The more specific accounting literature has thrown up a number of differences which result from experience and which are briefly summarised on pages 174-175 of McAulay et al (1998): "levels of self insight and the ability to form judgements which are likely to be consensual (refs), memory recall (refs), the sequencing of search for information (refs), and interest in inconsistencies (refs)".

A curious finding is that professionals are not necessarily the most effective decision makers. This follows from studies which address biases in the formation of judgement. I suppose it also ties in with ancient philosophies which argue along the lines of "out of the mouths of babies …." My daughters always seemed to me to be much more wise than me when they were teenagers! Now in their 20s, I have seen their wisdom and behavior regress.

Perhaps one or more of the following references will prove helpful to the authors. Apologies for not giving the full references - designed to save my fingers. Ashton and Kramer is clearly of obvious relevance:

Dreyfus and Dreyfus, 1986, Mind over Machine: the power of human intuition and expertise in the era of the computer, New York, Free Press. Dirsmith and Covaleski, 1985, Informal Communications …. Accounting Organisations and Society, 10, 146-159. Ashton and Kramer, 1980, Students as surrogates in behavioural research: some evidence, Journal of Accounting Research, Spring, 269-277. Frederick, 1991, Auditors' Representation and Retrieval of Internal Control Knowledge, Accounting Review, April 240-258. Bouwman, 1984, Expert versus Novice Decision Making in Accounting: a summary, Accounting, Organisations and Society, 9, 325- Tubbs, 1992, The Effect of Experience on the Auditors Organisation and Amount of Knowledge, The Accounting Review, October, 783-801 Libby and Trotman, 1993, The Review Process as a control for differential recall of evidence in auditor judgement, Accounting, Organisations and Society, 18, 559-574. McAulay, King and Carr, 1998, Expertise and learning … British Accounting Review, 30, 173-200.

Laurie McAulay University of Loughborough