Biological Observation Analysis and Repeatability

There has been considerable interest in the repeatability of scientific research. Ioannidis (2005) showed that most claimed research findings are false and Simmons et al. (2011) neatly demonstrate how simple methodological decisions made by an experimenter can increase the number of false-positive errors. Consider the case of an experimentalist who has various ‘degrees of freedom’ in the way that they conduct and analyse their experiment. These freedoms have a large influence on the conclusions draw from the results, particularly on the proportion of false-positive errors. For example, an experimenter can decide to increase the number of replicates if they do not have a significant result in their original experiment. They also have the flexibility to either include or exclude outlying values and set arbitrary thresholds for the inclusion of observations.

Similarly, observers of organisms have similar freedoms that contribute to their errors. For example, if an observer expects to find an organism in a habitat they may continue surveying only until they find those organisms that they are anticipating will be present. Furthermore, if they expect certain organisms they are perhaps more likely to misidentify similar looking organisms as the organism they expect to see, thus simultaneously creating a false-positive and a false-negative observation. There are many other observer behaviors that can lead to errors. Observers do not disclose their taxonomic biases, such as ignoring grasses and sedges. Observers will vary in their treatment of dead organisms, with some people treating them as a sign of occupancy and others not. Observers will vary in their methodology, either in what they accept as evidence of occupancy and by the equipment they use. Such choices introduce biases even when the observers are diligently trying to conduct a survey to the best of their ability. Though there are undoubtedly also cases where observers consciously manipulate their findings or carelessly report them wrongly (Sabbagh 2001; John et al., 2012).
The generation of false-positives among experimentalists also parallels biological observers in another characteristics. There is a well-known and widespread publication biases in the scientific literature towards positive results (Francis, 2012). Similarly, in my experience observers of biodiversity are much more likely to report observations of an unusual species, than ubiquitous species, particularly is that species is easily identifiable.

There are many ways that field recording could be improved, but we will always rely on cleaver analysis techniques to extract information from our data. The question is, can we peel off the layers of biases to ever know anything biologically relevant from our observations. In analysis of botanical records there is a tendency to include every observation, but if doing that only increases the biases, then ignoring biased observations can improve repeatability. It might seem rash to ignore data, but be skeptical of methods that use all the available data, particularly where the data is aggregated to disguise the biases.

References

Francis, G. (2012) Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19(2), 151-156. doi: 10.3758/s13423-012-0227-9

Ioannidis J.P.A. (2005) Why Most Published Research Findings Are False. PLoS Medicine 2(8), e124. doi:10.1371/journal.pmed.0020124.

John, L.K., Loewenstein, G., & Prelec, D. (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological science, 23(5), 524–532. doi: 10.1177/0956797611430953

Sabbagh, K. (2001) A Rum Affair. A True Story of Botanical Fraud. 1–276. Da Capo Press.

Simmons, J.P., Nelson, L.D. & Simonsohn, U. (2011) False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359–1366. doi: 10.1177/0956797611417632

This work by Quentin Groom is licensed under a Creative Commons Attribution 3.0 Unported License.