Excess Success in Investigations of Object-Based Attention

In this seminar Prof. Gregory Francis will discuss the replication crisis in psychology with specific examples taken from the literature on object-based attention.

In a seminal paper, Egly, Driver, and Rafal (1994) used the “two rectangles method” to demonstrate object-based attention. Their findings motivated hundreds of other studies that varied stimulus or task properties, thereby revealing detailed workings of visual attention.

This literature, however, shows statistical attributes that are sometimes related to questionable research practices that can undermine the reported conclusions. To quantitatively explore the prevalence of such attributes, Prof. Gregory Francis will present an application of the Test for Excess Success (TES) to the 37 papers were identified as investigation of object-based attention with four or more experiments.

The TES analysis estimates the probability that a direct replication of a set of studies would have the same success as the original report (or higher). If this probability falls below 0.1, it is typically considered unlikely that the original studies were conducted properly and reported completely.

The results showed that 19 of the 37 considered publications (51%) fall below this criterion. That is, the reported results in these publications seem too good to be true: even if the effects are real the studies lacked sufficient power to consistently produce successful outcomes.

Prof. Gregory Francis will also describe a new large sample study, which (similar to a previous large sample study) showed that the object-based attention effect depends on the orientation of the rectangles. This finding seems inconsistent with the theoretical interpretation of the two rectangles method for investigating object-based attention.