Teaching Research Methods
Much of students’ appreciation of science focuses on the results of scientific inquiry. They take the credibility of the inquiry on faith. Professional scientists do the same when they operate outside their domain of expertise, but within that domain they are supremely critical, rigorously questioning the means by which the results were obtained. They are very much aware of the fallibility of their investigatory tools and are on the lookout for artifacts and misleading results—findings that do not reflect the phenomenon under investigation but instead are due to limitations in the tools or design used to produce the results. An important part of learning a scientific field, therefore, is to come to appreciate and what its research methods are capable of showing as well as the various ways of avoiding artifacts or misleading results.
How We Currently Teach Research Methods
Certain topics occur in almost any course on scientific research methods, suggesting the existence of some core of basic principles. The design and interpretation of experiments is certain to be a focus, and nonexperimental methods such as naturalistic observation and correlational studies often are included as well. Students learn to define variables and distinguish the roles they can play in scientific investigation (manipulated, controlled, confounded, affected, hidden, associated, and so forth). Hypothesis testing is presented, and statistical tests or other means of assessing the reliability of findings are at least alluded to.
Although these topics can be taught in a very general way, leaving students free to apply them within any of the sciences, most departments offer their own, discipline-specific research methods or laboratory courses or incorporate methods within multiple content courses. By the end of their studies, psychology students will know how to manipulate at least two variables and use analysis of variance to interpret the resulting pattern of scores on a dependent variable such as reaction time. Biology students will know how to use certain laboratory instruments and techniques to obtain replicable findings concerning biological mechanisms, and some will know how to incorporate basic statistical analyses. Moving beyond experimental methodology, computer science students will know at least one programming language and appreciate its algorithmic structure. Linguistics students will have learned linguistic analysis within courses on syntax, phonology, and other core areas. Cultural anthropology students will know how to make textured observations using ethnographic methods. Finally, philosophy students will know how to construct and critique arguments and may have been exposed to the abstract characterization of confirmation and falsification offered within traditional philosophy of science.
A Special Challenge in Interdisciplinary Contexts: Orchestrating Multiple Methods
What distinguishes research in cognitive science from research in the various contributing disciplines is the demand to routinely engage different methodologies, often ones originating in very different disciplines. At the core of early cognitive science was the effort to employ computational modeling techniques (especially artificial intelligence programming) to account for behavioral data or competencies. As cognitive science has matured, the range of disciplinary methods that must be orchestrated has only expanded. The advent of functional neuroimaging, for example, requires the combined expertise of physicists skilled in understanding and measuring magnetic changes, haematologists who understand principles of blood circulation, neuroscientists who understand neuroanatomy and neurophysiology, psychologists skilled in developing behavioral tasks that tap selected cognitive capacities, statisticians who can develop novel tools for analyzing the vast amounts of data that are generated, as well as technicians who can calibrate the scanner. Indeed, Michael Posner was recruited to the Center for Higher Brain Function at Washington University because the grant by the James S. McDonnell Foundation establishing the Center required engagement of a psychologist. His expertise in designing behavioral tasks and in using the subtraction method to analyze the resulting reaction times proved critical in developing the related subtraction technique that underlies much neuroimaging research.
Integrating methods is not easy, however, even for established researchers. Generally a cognitive scientist is appointed in a traditional discipline and is expert in its methodology. By becoming conversant (preferably expert) in at least one other methodology, an individual gradually comes to appreciate the distinctive contributions and limitations of each method and can foresee ways the different methods may be related without inflating what each can provide. Sometimes alone and sometimes in collaboration with cognitive scientists based in other disciplines, a researcher who has gained these competencies can answer questions that will not yield to a single methodology. Thus, obtaining any finding in cognitive science rests upon knowledge of: (a) the basic principles that undergird scientific methods in any science; (b) specific methodologies from two or more disciplines; and (c) appropriate and effective orchestration of those methodologies.
How can such hard-won expertise be communicated to students? A realistic answer is that without intensive graduate-level education it cannot. However, for a good range of topics initial mastery is attainable—that is, knowing the material to some depth and being able to apply it to new situations, critique cases, etc. For other topics, we must be satisfied with showing students a glimpse of what is possible. In designing a cognitive science curriculum, one of the challenges is that these determinations must be made not just for a single body of material, but for material of all three types—(a) through (c) above—spanning multiple disciplines.
How We Currently Teach Research Methods in Cognitive Science
The difficulties and opportunities have been fingered in several reports. The Alfred P. Sloan Foundation, which underwrote much of the growth in cognitive science in the 1980s, also sponsored three conferences on education in cognitive science. More recently, an NSF planning workshop chaired by Neil Stillings in May 1993 produced “Undergraduate Education in Cognitive Science: Current Status and Future Prospects.” The following summary recommendations were made in this report:
Stronger agreement at the national level about the core content of the undergraduate major is needed. . . . In addition, a new round of funding from government and private foundations will be required. The funding will depend on a stronger recognition among cognitive scientists at a national level about what is needed and on the initiative of individuals and groups in making proposals for the development of curriculum and course materials. The highest priority should be on proposals to produce instructional materials that are widely usable.
More specifically, the report targeted the need for interdisciplinary methods courses and “courses and materials that incorporate new research developments in cognitive neuroscience, artificial neural networks, and situated action theory.” It noted that courses should teach “integratively rather than through the juxtaposition of traditional disciplinary courses” and should be “designed from scratch to embody the cognitive science approach. Appropriate textbooks, software packages, and other curricular materials for such courses are still scarce.” Finally, developing materials “that are exportable and that capture the special perspective of cognitive science requires faculty time and in some cases equipment, software, and technical or clerical assistance that is not normally available in program budgets.” This proposal is responsive to the report’s recommendations and to the need for special funding and effort that it highlighted.
Fifteen undergraduate cognitive science programs completed surveys used for this NSF report in 1993, and we recently examined the websites of a similar range of programs in order to achieve an updated characterization of the current status of education in cognitive science methodology. Like cognitive science itself, the programs are diverse. Some focus on a particular part of cognitive science, often involving just two disciplines (e.g., the Linguistics and Philosophy major at MIT, the Neurosciences and Behavior certificate at University of Colorado, and the Biopsychology certificate at Georgia Institute of Technology). Most of the broader programs can be identified with one of two basic formats:
1. Cognitive science in depth. This format is common in universities that have granted full departmental status to cognitive science (Brown, Johns Hopkins, MIT, Rochester, and UCSD) or where cognitive science is available as a primary major (e.g., Northwestern, Pennsylvania). Following an “Introduction to Cognitive Science” or a sequence of initial courses that combine content and method for several topics (e.g., language, perception), students take separate, rigorous courses in the methods of several disciplines. Later they use those methods in advanced courses such as artificial intelligence or cognitive neuroscience. For example, majors at Northwestern University take Comp Sci 110 (computer programming), Psych 201 (statistics) and Psych 205 (research methods), and depending on choices among core courses may also acquire introductory knowledge (including methods) of linguistics, neuroscience, and computer modeling. All majors at University of Pennsylvania take formal logic, two computer programming courses, and in other core courses acquire introductory knowledge (including methods) of linguistics, neuroscience, philosophy, and experimental psychology. (Note that a few highly-regarded, in-depth programs deviate significantly from this format but share the characteristic of having no overall methods course; the best example is the unusually flexible program in the School of Cognitive Science at Hampshire College.)
2. Cognitive science as an adjunct to other studies. Many nondepartmental programs
follow a format requiring fewer courses overall, fewer courses in methodology,
and fewer specially-designed interdisciplinary courses that integrate theory,
findings, and methods. Often the credential offered is a second major, minor,
or certificate. Although this may appear inferior to the first format, far more
students are able to fit this kind of major into their educational program.
Typically they obtain depth in at least one of the contributing disciplines
by completing an additional major in a traditional department. Examples include
the cognitive science minor at University of Alabama-Birmingham, Illinois State
University, Ohio State University, and our own second major in Philosophy-Neuroscience-Psychology.
Towards a New Way of Teaching Research Methods in Cognitive Science
As a preliminary to designing our own course, we have identified two programs that do not fit either format and include a specially-designed interdisciplinary methodology course in their curriculum (in addition to emphasizing methodology in some of their other courses). Vassar offers an unusually well-integrated core of three courses that combine content with methodology in specific areas plus “Research Methods in Cognitive Science,” which applies multiple methods to a single problem such as sentence processing. Also required are a traditional statistics and research design course offered by the psychology department, four electives, and a senior thesis. Indiana University's cognitive science majors take four specially-designed courses: Philosophical Foundations of the C&IS, Mathematics and Logic for C&IS, Computation in the C&IS, and Experiments and Models in Cognition (C&IS=“Cognitive and Information Sciences”). They also obtain breadth by taking an introductory course in each of four departments and depth by taking three related courses (e.g. neuroscience courses). Also of interest is the laboratory component of a philosophy course, Minds and Brains, that is an elective in the neuroscience major at Trinity College. The instructor, Dan Lloyd, will serve on our National Advisory Board.
There is much to admire in the methods courses at Vassar and Indiana. First, they were specially designed for cognitive science programs and therefore can efficiently provide the kind of window on methods that such programs wish to provide to their students. Second, they address all three aspects we identified above as important in an interdisciplinary methods course: (a) basic principles of research design, (b) discipline-specific methods, and (c) orchestration of multiple methods. Third, they teach the methods in the context of considering specific cases (case-based learning). Fourth, they engage students actively in hands-on activities. We propose to develop a course design and course materials that embody all of those virtues, and three others as well, namely: modular design; access to materials in a low-friction, low-cost medium (the web); and making maximal use of a single interdisciplinary methods course for students taking a less extensive cognitive science curriculum. This is a major undertaking, but the time is right in terms of demand and available technology.