In Chapter 2, we made the distinction between in vivo and in vitro methods for studying science. To review, the former involves actual case-studies of scientific practice; the latter involves experiments that simulate aspects of scientific thinking.
The advantages of experiments come at a cost in terms of realism. A simulation can and should model only certain aspects of science, and therefore simulations need to be complemented by case-studies; experiments can suggest features that ought to be studied in cases, and cases can suggest variables that ought to be manipulated in simulations. Psychologist prefer experiments, in part because they want to appear to be scientific . Sociologists of scientific knowledge generally prefer 'thick descriptions' of actual cases. The obvious answer is we need both--and we need to connect these two, using experiments to isolate variables that seem to play a role in cases, then going back to do further case-studies looking, in part, for the kinds of patterns suggested by experiments .
Experiments and case-studies can also be used in the classroom to teach scientific thinking. For example, I have used both the 2-4-6 task and a task based on the card game Eleusis to illustrate what is meant by verification and falsification, and how they work in practice . I ask students to work in groups to try to solve rules like ‘three different numbers’. On this particular rule, it is easy for students to imagine positive tests, because virtually every triple they try initially has number patterns that go up or down. I have them write down their hypotheses so when they are done we can talk about confirmation and disconfirmation. In discussion, I try to get them to identify examples of positive and negative tests from their own experiments, and get them to discover that whether a test is confirmatory or disconfirmatory depends on their hypothesis and how they represent the rule.
Once students have these analytic tools under their belts, they can apply them to notebooks or other detailed descriptions of invention or discovery . In the course of applying what they learned from experiments, students will also come to see the inadequacies of models derived from simulations. For example, I like to lead them on a discussion of how the 2-4-6 task does and does not model scientific practice, and how it could be made more ecologically valid .Then I can introduce the version with error, in order to show how one can add features that simulate additional aspects of real-world science.
Robert Rosenwein and I have proposed a complex simulation based on William Gamson’s "Simulated Society" (SIMSOC). In SIMSOC, class participants are placed in one of four regions, given unequal amounts of "Simbucks" and compete to get ‘simjobs" in simulated industries or political parties or media a kind of judicial group.
Bob and I proposed to convert SIMSOC into a SIMSCI, which would substitute scientific laboratories for industries, different theoretical perspectives for political ones, publication outlets for media and granting agencies for judicial. Each participant would be assigned to one of several research teams, except for a handful, who would be left to work on their own or join a team. Teams would have to choose problems, compete for resources, and publish brief accounts of their work.
Teams would start with unequal resources, in terms of 'Simbucks' and equipment. For example, one research team might have a computer that allowed them to work on an artificial universe task like the one created by Mynatt, Doherty & Tweney. Other teams would have to pay for access to this equipment. Because experiments were costly, research teams would have to spend their resources carefully.
One of the ways of attracting more resources would be to announce a new discovery and have it substantiated, thereby increasing the likelihood that the funding agencies that are part of SIMSCI would support the team's research. A wide range of task variables could be manipulated to increase or decrease the likelihood of a discovery, including amount and type of error present in the problem.
This SIMSCI environment could potentially serve both research and pedagogical aims--it could be used to test the way in which a number of variables affect problem-choice and discovery, and also teach students about the way in which scientific thinking is embedded in a network of social negotiations.
The biggest advantage of in vitro simulations is that they allow the researcher or educator to manipulate variables that might affect discovery--like the amount of error present in a particular task, the distribution of resources, the agendas of funding agencies, and the like. SIMSCI could include the manipulable features listed below:
1. Resource allocation and control:
Scientists--especially senior scientists--spend an inordinate amount of time seeking funding for their laboratories. The effect of this struggle for resources could be simulated by including funding agencies and laboratories in SIMSCI. Resources could be located in laboratories, each of which would start with a director who would be responsible for keeping the lab alive by recruiting new members and obtaining further funding. Lab directors would begin with unequal resources, so that some would have great advantages over the others. Some participants would belong to no lab, and could be recruited to assist in research.
To get additional resources, labs could appeal to two funding agencies, one of which worked like a federal agency, with proposal criteria and rules that indicated funding priorities and a review process that included student participants as reviewers. Another funding agency might have a tacit agenda, simulating foundations like MacArthur that decide on winners without a standard application process. Labs that managed to guess the tacit agenda would be given funding. It would be interesting to see if the peer-refereed funding agency began to reward the same winners, thus creating a Matthew effect.
2. Nature of the Task:
Another manipulable feature of Sim SCI will be the types of tasks participants could choose to work on. We want these tasks to simulate aspects of scientific thinking; fortunately, there is a long cognitive literature on such tasks, including ones that use numbers, or cards, or programmable devices, or complex artificial universes .
Experimental: This sort of task involves generating repeated manipulations of a phenomenon in an effort to discover and test hypotheses. Most of the tasks used in the literature on scientific reasoning are of this sort. For example, one could ask students to determine the rule that dictates how playing cards can be laid out in a series. To determine the rule, students would have to try different card sequences, and receive feedback on whether each was correct or incorrect.
(b) Observational: This sort of a task involves carefully watching a process without influencing it--except that the act of observation can itself be an influence. Tasks of this sort are less common in the psychology of science literature. We outline one which resembles a problem in astronomy below.
Tasks would require different resources. For example, a simple number problem like the 2-4-6 task might be relatively cheap--pencil and paper and access to a calculator that could give feedback on trials. An observational task, in contrast, might require a computer with a color monitor, access to which could only be purchased by a lab. Problem choice would be dictated, in part, by resources, but discoveries could be made on inexpensive problems. It would be interesting to see whether these inexpensive solutions are assigned less status than solutions to more expensive, resource-intensive problems.
3. Communication:
Another important, related issue is communication. SIMSCI needs to be able to model sharing of scientific information and the ‘publish or perish’ structure of the scientific reward system, including the emphasis on achieving priority.
(a) Journals:
Obviously, a time-limited simulation cannot simulate all the rhetorical features of actual science. But one can include competing newsletters--perhaps electronic ones, disseminated via e-mail--which report brief accounts of results and will be peer reviewed by students. Initially, newsletters will be based in competing labs with extensive resources, but any group of participants that can assemble sufficient resources will be able to start its own newsletter. Subscriptions will cost resources, which will be turned over to the newsletter. So again, an underfunded newsletter can gain resources by attracting subscribers.
(b) Conference presentations:
Just as any group or lab can organize a newsletter, so any group can organize a conference. But participants will have to spend resources to get to the conference. One alternative will be to be invited to a conference supported by a lab, or be sent by a lab to a conference. In both of these cases, the lab would have to pay.
(c) Informal contacts:
Participants will also be able to send written messages to other students whom they can identify by name and role. (This could be done via e-mail). This simulates the kind of informal communication that takes place in letters. The receiver of a message will, of course, be under no obligation to respond.
Again, constraints on all these forms of communication could be manipulated in a variety of ways in SIMSCI. One could, for example, have newsletters reward positive results and refuse to publish replications--or one could do the reverse. One could even attempt to assess the importance of rewarding priority of science by providing no reward for it. What if multiple, independent re-discoveries were given equal status? How could one be sure what counted as independent? The point is, SIMSCI does not have to mimic the structure of science as it usually appears--one can deliberately experiment with different kinds of reward structures.
4. Outside Events:
SIMSCI will also allow the controllers to introduce a wide range of outside events. For example, at a certain stage of the simulation, improved equipment could be made available which would provide higher resolution on the perceptual task, potentially making terrain or other features far less ambiguous. Use of this new technology might be very expensive. One might also introduce the possibility of 'lower tech' improvements that could be made and/or utilized by lower resource groups. Using mechanisms like this, SIMSCI could begin to explore the role of technological improvements in science.
One could also introduce a new theory, or task, or change the priorities of one of the funding agencies. Special awards like the Nobel Prize could be introduced. Controllers could select outside events that reflect the goals of their simulation.
SIMSCI could include a variety of measures of the cognitive and social processes of its participants, ones that they could use to reflect on and improve their own efforts to solve problems and succeed as scientists and that could also be used to compare permutations of variables like differences in resources and changes in reward structure. The SIMSCI manual would include suggestions on how to get the maximum amount of information out of the minimum amount of data.
(1) Documents:
Each participant in SIMSCI will be required to keep a notebook, recording ideas for experiments and actual results. Participants could be shown portions of Faraday's notebook and Alexander Graham Bell’s as examples.
Participants would also be writing e-mail messages and longer articles to appear in electronic newsletters. In addition, participants would be writing proposals to funding agencies. All of these public documents, and all drafts of them, would provide a useful record for further reflection, including those articles that were not accepted and proposals that were not funded.
(2) Protocols:
Selected participants could be protocoled as they work on their tasks, i.e., they would be asked to talk aloud as they work. Laboratory meetings could be taped, creating a record similar to the one produced by Kevin Dunbar’s molecular biology laboratories . Conferences could be videotaped. Meetings of gatekeepers--journal editors, foundation boards, review panels--could be recorded as well.
(3) Interviews:
Selected participants could also be drawn aside for interviews at random times during the simulation. The interviews will allow exploration of cognitive/social relationships--they could be asked questions about their progress in solving the tasks as well as their career trajectories and their attitudes toward funding agencies, laboratories, etc.
These measures will reveal a variety of processes used by groups and in response to the simulation, responses that could be used by participants and researchers afterwards to analyze what happened in a particular version of SIMSCI. Probably the result would be more provocative new questions than answers; some of these questions would certainly lead to new case-studies. Experimental simulations have provided frameworks for the study of scientists like Faraday and inventors like Bell . Similarly, SIMSCI could generate frameworks for studying how cognitive and social factors interact in scientific groups.
To better understand the strengths and weaknesses of a complex in vitro simulation, let us consider an example of a SIMSCI focused on a complex issue: evidence ambiguity. Evidence is rarely unambiguous; debates in science often revolve around what constitutes 'good' data and what should be dismissed as error. SIMSCI allows us to manipulate different types of error and note their effect on consensus formation and on the construction of order in domains where the level of randomness is high.
(1) Error on an experimental task:
In Chapter 2, I discussed attempts to simulate possible and actual error in psychology experiments. Suppose the 2-4-6 task were one of the problems participants could elect to work on. To conduct an experiment, each student would have to pay a nominal fee to get access to a calculator or computer, which would give them the result. One could add a high level of error to this task--say, 40%. One could add the possibility of paying to use better equipment in order to reduce the possibility of error. Participants would then have to decide whether to spend more on a few good experiments, or run lots of relatively cheap ones, replicating to check for errors.
Again, laboratories with more resources will have substantial advantages on such a task, though those with less resources will still be able to make substantial progress if they design their less expensive experiments cleverly.
High levels of ambiguity, or error, also give more room for participants ton construct rules and negotiate what constitutes progress on the task. Rules could be complex enough to allow for different hypotheses and constructions, especially on more complex tasks like the artificial universes created by Mynatt and his colleagues . One could even introduce a very complex task that had no rule, to see what rules participants would invent, especially if they suspected there was a lot of error in the task. SIMSCI therefore allows one to explore the relationship between resources, rule ambiguity, level of error in the data and research strategy.
(2) Error on an observational task:
This sort of a task often involves resolving perceptual ambiguity through careful observation. Examples include the relationship among geological strata, or resolution of terrain features on a distant planet.
Consider, for example, a SIMSCI task based on the controversy concerning the canals on Mars. Percival Lowell developed a theory and a set of supporting observations concerning the presence of canals on Mars. These canals were seen by other prominent astronomers, but still others remained skeptical. Interestingly, several critics conducted experiments to determine whether the canals might be an optical illusion, illustrating that even practicing scientists can turn to simulations to settle controversies. Eventually, more powerful telescopes established that there were no canals, but until the advent of this new technology, the controversy raged .
Participants could be allowed to select computer screens representing satellite images of terrain features on a distant planet; they would be told that the satellite is not functioning well, and therefore the resolution of the images is poor. Again, access to the computer screens will require commitment of resources; well-funded labs will have considerable advantages.
In addition, information on conflicting hypotheses could be made available to participants, along with evidence that proponents claim support each hypothesis. One hypothesis, for example, might argue that there were terrain features which suggested the presence of intelligent life; another might argue that these were all just natural features. The papers describing these hypotheses could originate from competing laboratories. At the beginning of this SIMSCI, each lab could be assigned a participant director who would have to decide whether to continue to support the lab's past position, or take a different tack. The risk of change would be losing funding from stable sources dedicated to a particular theory. For example, one of the funding sources might represent a space agency that wanted proof of intelligent life on other worlds. A lab that stuck with that agenda would increase its chances for funding.
One could use this sort of a SIMSCI to study under what circumstances data plays the largest and smallest role in determining the outcome of scientific discoveries. The SIMSCI might begin with enough ambiguity that teams could argue persuasively for their perspectives, but gradually introduce new equipment that gradually improved the quality of the data. One could vary the extent to which this improved data pointed towards a genuine discovery or simply showed no evidence of a pattern. One could even try to create paradigm shifts by introducing anomalous results. Then one could watch the negotiations that went on between groups.
The kind of deep commitments to research programs seen in science are hard to simulate, but social influence studies like Zimbardo's prison simulation and Milgram's obedience experiments show that subjects can quickly identify themselves with arbitrary roles assigned by an experimenter. We might be surprised at the extent to which participants in SIMSCI become committed to different theoretical positions.
These three main manipulable domains and the manipulable features within them will allow SIMSCI to be adapted to explore relationships between social and cognitive variables. For example, one could study minority influence in science by introducing a trained confederate into the simulation, who would vigorously and persuasively promote a hypothesis at variance with the positions taken by the dominant labs on a particular task. One could train this confederate to adopt specific persuasion strategies to see which worked best.
Similarly, one could introduce a confederate who deliberately used fraud in an effort to bolster his/her career. (Note that SIMSCI does not automatically exclude the possibility of fraud--a student can always misrepresent the results he or she has actually achieved, and others will have to check by replicating).
I have outlined SIMSCI in a way that makes it ambiguous whether it is primarily a research platform or a teaching device. That ambiguity is deliberate. Clearly, this kind of a simulation, properly done, could teach students a lot about the way social and cognitive aspects of science blend in actual practice. It could also allow researchers to manipulate and measure factors that might affect the resolution of scientific controversies, as well as simulate different models of scientific progress. I think the research/teaching dichotomy should be transcended, whenever possible. A SIMSCI could be set-up in a class, such as the one I teach on scientific and technological thinking, and the student participants could be major players in conducting a post-mortem that would evaluate what we could learn and generalize from the experience. For example, student participants could discuss the conditions that promoted discovery and creativity, and those that did not.
Science education rarely includes much about the relationships between cognitive and social factors in science; typically, students learn about the context of science in separate courses on history and/or philosophy of science SIMSCI could complement these courses by making students active participants in science, allowing them to experience a small part of the joys and frustrations of a scientific career. Indeed, SIMSCI could be incorporated into a variety of such courses, and the tasks used in the SIMSCI environment could be linked to, and enriched by, a variety of case-studies.
SIMSCI need not be limited to science and engineering students. Indeed, it is a perfect platform for teaching non-science students how science really works. SIMSCI can become a platform for studying issues in science policy and education; it can be used in classrooms and research laboratories. It cannot replace case-studies and thick descriptions; instead, it would complement these approaches. One could, for example, take the approach advocated by Dunbar and iterate between a study of an actual scientific laboratory and a SIMSCI that allowed manipulation of variables that appeared to affect consensus formation in that lab. Data collected from SIMSCI could help explain the patterns of response seen in the actual laboratory and also suggest surprising new relationships to look for in case studies. SIMSCI could also complement historical case studies, as the canals on Mars example suggests, and computational simulations modeling the effect of the same variables on a multi-agent network.
Simulations like Civilization and SimCity suggest that a computer SIMSCI could be created and run over the internet. Consider Civilization. In this simulation, or game, one plays the role of a civilization-builder, from 4000 B.C. to the present, making all decisions about where to build cities, what structures to construct in them, which technologies to create and what relations to have with other civilizations. One has to maintain a simple economy, balancing taxes with expenditures and providing luxuries to keep people happy. One can compete with other computerized opponents or human opponents over the internet. Judging from the number of Web-sites and books of hints available for Civilization II, this kind of simulation is engrossing to the point of addiction.
In a computerized SIMSCI, the virtual world of laboratories, tasks and simbucks could be enhanced by graphics and other features that motivate players to spend hours mastering Civilization. For example, just as one has to accumulate resources to pay for civilization advances in Civilization and urban improvements in SimCity, one could pay to improve laboratories in SIMSCI, buying research equipment, technicians, and even try to entice top-level researchers to leave others’ labs and join yours. Part of a laboratory’s income could come from users outside of the lab who pay for the use of its equipment. Labs could compete to offer the best facilities and services, while also competing for grants.
In Civilization, one simply buys technological advances and scientific discoveries. In a computerized SIMSCI, funding from foundations and agencies could depend, in part, on results achieved with the equipment--on discoveries and accomplishments. The funding agencies could be represented by internet participants assigned those roles. Newsletter journals containing results and theories could be formed and distributed over the internet. Laboratories would have to gain reputations for expertise and quality. One could even simulate the kinds of negotiations that lead laboratories to collaborate on ‘Big Science’ projects like the Superconducting Supercollider, and introduce events that could affect their decision to stay with or abandon such projects.
In this manner, a computerized SIMSCI could simulate the complex relationships between basic and applied research and technological innovation. For example, Stokes talks about what he calls Pasteur’s quadrant. If one thinks of benefit to basic science as one axis on a graph and applied benefits as another, then the quadrant corresponding to high potential for both is the area in which Pasteur worked: he created the field of microbiology and at the same time his work had immediate pay-off for brewers One could set up the funding agencies in a SIMSCI so that one encouraged basic research and another applied, and see if a lab emerged that could connect the two. One could also give a lab a ‘work-in-Pasteur’s-quadrant’ heuristic and see how it managed to translate this idea into action.
One of the advantages of a computer SIMSCI is that the participants could actually create highly complex technologies, like a new weapons system. This capability could be used to set-up ethical dilemmas. Should the ‘work-in-Pasteur’s-quadrant’ heuristic lab use creating weapons as a justification for its basic research?
From an educational standpoint, these computer simulations are highly motivating--indeed, one would have to be careful that students not spend too much time on them! From a research standpoint, they are highly manipulable, if programmed properly--one can introduce all kinds of contingencies as the simulation progresses.
One could, for example, introduce ethical issues. The possibility of producing fraudulent data and publishing it is always present. One could simply watch to see if this ever happened, and note the circumstances. Or one or two students could be given instructions to engage in forms of fraud, to see how the system responded. Similar opportunities exist for conflicts of interest, e.g., sitting on a panel that reviews a proposal from a competing lab.
One of the advantages of Civilization II is that participants can buy sustainable technologies like recycling and solar power, which helps them avoid local pollution effects. It is probably too much of a stretch to simulate this kind of sustainable technology in SIMSCI, though one might give participants the option of choosing a problem that promised to pave the way for sustainable technologies. Then one could experiment with how such attempts fit in with the shifting agendas of funding agencies, perhaps contrasting it with scientific developments supported by a military agency.
The point is, the possibilities are endless. Even just having a class try to design a SIMSCI would be a useful exercise in thinking about how science really operates.
This page was last edited: Wednesday, July 14, 1999