It becomes clear, looking at these examples, that some correlations are indeed only false positives, or in the language of statistics „Type I errors”. They are likely artifacts of sampling error, i.e. the fact that we deal with limited samples of the phenomena we try to study. What we see in these samples can sometimes deviate very much from what is really going on in the bigger picture.
In 2017, researchers from the Eötvös Loránd University in Budapest observed a positive correlation between the recall performance of domestic dogs on a novel command learning task and the number of sleep spindles – brief bursts of brain activity recorded with EEG from the animals during sleep. „This was an exciting cross-species replication of earlier observations in humans, rats, and mice, but we needed more certainty.” Says Vivien Reicher, co-leading author of the study. „The problem is that non-invasive research, the standard in work with companion dogs, cannot demonstrate the suspected causal relationships directly, while this effect is also questioned by some researchers in the field.”.
Indeed the largest to date study in humans, with over 900 subjects, could not confirm that sleep spindles correlate with learning. To complicate things, however, studies in slice preparations and also in living mice had uncovered potential mechanisms and demonstrated causal dependence where work in humans and dogs could only scratch the surface. „Nonetheless, we saw ourselves in the position to make an important contribution to the controversy through our work in the dog, since we had completed two more learning experiments following a very similar set-up as our first publication on dog sleep spindles in 2017.”, says Anna Kis, senior co-author which developed the learning paradigm back in 2016. The researchers then set out to analyze all related data sets with the same detection algorithm, developed back in 2017. „This was a crucial aspect of our contribution,” says corresponding author Ivaylo Iotchev, „different automatic detectors for sleep spindles were shown to diverge in their conclusions about the number of sleep spindles in a sample of EEG. Thus for our replication analysis, it was very crucial to use the same detection method.”
The result was not a straightforward yes or no, instead, the prevalence of significant positive correlations between sleep spindle occurrence and learning was higher when individual measurements of each were averaged and only then compared. „Sleep spindles were measured on three occasions in each data set”, says Ivaylo Iotchev, „we also had tested learning performance twice in the two newer studies. We were interested in these averages because random measurement error can be expected to decline with averaging. Measurement error means, that the individual circumstances of each measurement can obscure the underlying relationship we try to study. Therefore the ability to look at averaged values for spindle occurrence and learning allowed us to reach one conclusion with fairly high certainty – the often reported positive correlations are not artifacts of measurement error, as they become more prevalent when an error is reduced.”
When judging the trustworthiness of correlations, researchers have to be alert not only towards Type I errors (i.e. false positives), but also Type II errors (or misses). Work in larger samples is currently still rare in research on sleep-dependent learning, but it will be necessary for establishing if the positive correlations observed between sleep spindles and learning are Type I errors or if failures of replication are merely Type II errors. This new finding in the dog adds to arguments for the latter.
Original publication: Iotchev, I.B., Reicher, V., Kovács, E. et al. Averaging sleep spindle occurrence in dogs predicts learning performance better than single measures. Sci Rep 10, 22461 (2020). https://doi.org/10.1038/s41598-020-80417-8
