By Dr. Rebecca Willén, founder of IGDORE
In a recent episode of Everything Hertz, Radical Transparency, I was interviewed by Dr. Dan Quintana and Dr. James Heathers about the retroactive disclosure statements I made public in February 2018 (also see the post I wrote on Twitter). The below text follows up on the interview and supplements it.
What did I do?
We’re many researchers who admit to have employed questionable research practices, such as p-hacking and HARKing. However, few make it explicit which particular questionable practice they employed in which particular study. People seem to be generally okay with making admissions on a general level, but less inclined to take it to a specific level. Dana Carney’s retroactive disclosure statement is a rare exception. Her bold initiative is what inspired me to do the same. I extended Dana Carney’s practice by writing a disclosure statement not only for one of my studies, but for each of my published studies.
Why did I do it?
Because it will make the past more useful. I believe that we in ten years from now will consider research published without an explicit disclosure statement (e.g. the 21 word solution) as exploratory by default. I find this very unsatisfactory for at least three reasons. First, truly confirmatory studies will be inaccurately interpreted as exploratory. Second, truly confirmatory hypotheses will be inaccurately interpreted as exploratory. Third, we will not have all relevant details to properly understand the methods and findings and build upon them.
An example of the latter is a study of mine from 2013. Simply viewing it as exploratory, without any further knowledge, wouldn’t have been of any help to my colleague in understanding why he wasn’t able to replicate our findings on one particular outcome measure. From my retroactive disclosure statement he learned that the outcome measure in question was actually a finding from exploring the data, in contrast to the other outcome measure that had indeed been planned from start.
I argue that it’s important to write and publish disclosure statements retroactively for all types of research in psychology; not only for research where questionable practices were employed. Indeed will this likely be the only way to save published confirmatory research (that does not already have a disclosure statement) from in the future being interpreted as exploratory and possibly questionable.
What do my co-authors think about it?
Four of my disclosure statements were published as part of my PhD thesis and have thereby been reviewed and accepted by my co-authors before publication. There are yet a few disclosure statements to write and they will be written in collaboration with my co-authors. That said, I do argue that every author on a paper has a responsibility to disclose scientific misconduct they know have occurred. This moral responsibility toward the scientific community and society as a whole is far more important than asking any co-authors for approval to publicly report previously undisclosed details about a published study’s methods or results.
What will the journals think about it?
This is indeed a very important question. My personal opinion is that articles where questionable practices were employed should not be retracted unless outright fraud* occurred. Instead of retractions should such articles (on the journals’ websites) be supplemented with retroactive disclosure statements such as Dana Carney’s and my own. My main objection against retraction in these cases is that the articles do contain information valuable for knowledge accumulation** — as long as they are supplemented with (retroactive) disclosure statements. Let’s not throw away information. Let’s instead update it and continue to build upon it.
Furthermore, retractions will deter researchers from being honest about their past actions. This will in turn make past research less useful than it can be. Another objection I have is that retractions will never happen in all cases where questionable practices were employed, and I don’t think it’s the worst or most important articles that would be retracted; instead the retracted papers will be the ones on hyped topics (e.g. power posing), by famous researchers or papers that for other reasons gain someone’s attention (e.g. in the case of Brian Wansink who naively wrote a blog post describing questionable practices being used in his lab).
How journals, funding agencies and employers decide to handle cases of disclosed questionable research practices will have a huge impact on the extent to which we will be able to gain clarity about potential misconduct in already published research. This will therefore be the main topic during our Open Science Meetup on Bali, 23–29 April, 2018 (Science in Transit: How to Deal with Past Sins when Science Opens Up?). The outcome of these discussions will hopefully result in a joint paper with recommendations to leaders in academia on how to handle cases of disclosed questionable research practices.
What could I have done better?
The disclosure statements I wrote in my PhD thesis were less explicit than the later ones. For example, in the thesis I never used words like p-hacking or HARKing; instead I simply described what we had done. This less direct approach does fulfil its purpose (i.e. to disclose), but to be frank: few will actually remember what it said (“did she p-hack or not? I’m not sure, but she did write something about using this analysis instead of that analysis so I think maybe she did…”). I would recommend everyone who dare to be frank to actually be it. Use the words. Do say that you p-hacked if that’s what you did. Others will read your statement and think “wow, s/he dares to say it, then I will too!”. In particular scientists with an already good open science reputation have little to lose and can become important role models in this regard.
I think we can improve several things in future retroactive disclosure statements, for example the following:
- When research has been reported as confirmatory, it might be more ideal to structure the disclosure statements as a list of the hypotheses, providing the previously undisclosed information for each one of the hypotheses. This would make it much clearer to which extent a particular finding is based on solid practices.
- It would be great to make disclosure statements dynamic (living) documents that can be independently edited by all authors on a paper, and that this document has a version control system running in the background. This way we would be able to add information with time, and any disagreements between authors could be viewed publicly in the document.
- Dr. Etienne LeBel has worked on an icon for disclosure statements, which could also be employed when the statements are made retroactively. I used colour codes in my list of publications to indicate whether everything had been reported transparently in the original paper; standardised icons would fill this purpose even better. Etienne has also suggested creating a template for the retroactive disclosure statements. I hope we’ll be hearing more soon about the icon and template.
*The implementation of explicit disclosure statements (e.g. the 21 word solution) at the point of article submission will soon result in many questionable research practices being considered outright fraud due to the fact that researchers must become explicitly dishonest to hide such practices. However, research published without such explicit statements should not be judged as harshly, because implicit dishonesty is not as severe as explicit dishonesty.
**Thanks to Daniel Berntsson.
