Retroactive Disclosure Statements Make the Past More Useful

By Dr. Rebecca Willén, founder of IGDORE

In a recent episode of Everything Hertz, Radical Transparency, I was interviewed by Dr. Dan Quintana and Dr. James Heathers about the retroactive disclosure statements I made public in February 2018 (also see the post I wrote on Twitter). The below text follows up on the interview and supplements it.

What did I do?

Why did I do it?

An example of the latter is a study of mine from 2013. Simply viewing it as exploratory, without any further knowledge, wouldn’t have been of any help to my colleague in understanding why he wasn’t able to replicate our findings on one particular outcome measure. From my retroactive disclosure statement he learned that the outcome measure in question was actually a finding from exploring the data, in contrast to the other outcome measure that had indeed been planned from start.

I argue that it’s important to write and publish disclosure statements retroactively for all types of research in psychology; not only for research where questionable practices were employed. Indeed will this likely be the only way to save published confirmatory research (that does not already have a disclosure statement) from in the future being interpreted as exploratory and possibly questionable.

What do my co-authors think about it?

What will the journals think about it?

Furthermore, retractions will deter researchers from being honest about their past actions. This will in turn make past research less useful than it can be. Another objection I have is that retractions will never happen in all cases where questionable practices were employed, and I don’t think it’s the worst or most important articles that would be retracted; instead the retracted papers will be the ones on hyped topics (e.g. power posing), by famous researchers or papers that for other reasons gain someone’s attention (e.g. in the case of Brian Wansink who naively wrote a blog post describing questionable practices being used in his lab).

How journals, funding agencies and employers decide to handle cases of disclosed questionable research practices will have a huge impact on the extent to which we will be able to gain clarity about potential misconduct in already published research. This will therefore be the main topic during our Open Science Meetup on Bali, 23–29 April, 2018 (Science in Transit: How to Deal with Past Sins when Science Opens Up?). The outcome of these discussions will hopefully result in a joint paper with recommendations to leaders in academia on how to handle cases of disclosed questionable research practices.

What could I have done better?

I think we can improve several things in future retroactive disclosure statements, for example the following:

  • When research has been reported as confirmatory, it might be more ideal to structure the disclosure statements as a list of the hypotheses, providing the previously undisclosed information for each one of the hypotheses. This would make it much clearer to which extent a particular finding is based on solid practices.
  • It would be great to make disclosure statements dynamic (living) documents that can be independently edited by all authors on a paper, and that this document has a version control system running in the background. This way we would be able to add information with time, and any disagreements between authors could be viewed publicly in the document.
  • Dr. Etienne LeBel has worked on an icon for disclosure statements, which could also be employed when the statements are made retroactively. I used colour codes in my list of publications to indicate whether everything had been reported transparently in the original paper; standardised icons would fill this purpose even better. Etienne has also suggested creating a template for the retroactive disclosure statements. I hope we’ll be hearing more soon about the icon and template.

*The implementation of explicit disclosure statements (e.g. the 21 word solution) at the point of article submission will soon result in many questionable research practices being considered outright fraud due to the fact that researchers must become explicitly dishonest to hide such practices. However, research published without such explicit statements should not be judged as harshly, because implicit dishonesty is not as severe as explicit dishonesty.

**Thanks to Daniel Berntsson.

IGDORE is an independent cross-disciplinary research institute. Check out our website for more!