Peer Review Week 2021 takes place September 20-24, with this year’s featured theme — “Identity in Peer Review” — dedicated to exploring the multifaceted nature of identity, how personal and social identity affects peer review practices and experiences, and what’s needed to foster more diverse, equitable, and inclusive peer review processes.
To mark the occasion, Dr Mario Malički, Co-Editor-in-Chief of Research Integrity and Peer Review, shares his thoughts and questions regarding identity in peer review.
Supporting diversity
As an editor, I would like to create and support an inclusive culture of peer review.
I must start with declaring my scholarly research focuses on peer review, and not on identity, diversity or inclusion. In some of my published studies, analyses my colleagues and I conducted explored associations of outcomes or modification of effects with career-stage, gender, statistical expertise, or country or state of individuals – but that is as far as my research or expertise on those topics go. As an editor, I want to create and support an inclusive culture of peer review. I am, however, not sure how authors, readers, reviewers, and the wider public and scientific community would like our or other journals to demonstrate or implement diversity practices. This means, as one of the steps, I must plan surveys and conversations with interested parties to find this out (and this blog is a part of that strategy). I apologise in advance for asking many questions in the paragraphs below and not providing answers to them. I hope, though, they will inspire sharing of experiences.
What should journals aim for and measure in regards to identity in peer review?
Previous cases have shown cases of suspicious self-citation or citation stacking patterns of journals – how do we ensure this does not happen to diversity indicators?
Are there specific quotas of representativeness journals should be aiming for when it comes to their peer reviewers, e.g., specific ratio of gender or sexual identity, career stages, incomes, institutions, countries or states, religions, races, (sub)cultures, multidisciplinarities, disabilities, etc.? (For example, see Lancet’s commitment to have 35% women reviewers, and 50% women in editorial boards, BMJ’s and Horizon 2020 Advisory Groups commitment of gender equity, and an analysis of diversity in USA academia).
How should editors help achieve those goals? Should they find reviewers that check several of the diversity identifiers at the same time? Should they make sure that the reviewers don’t feel they are primarily being invited to fulfil a quota? Should editors have a list of reviewers that are happy to be invited to fulfil those quotas? Can editors be sure that if they collect this information from reviewers (or authors and editorial board members), that their publishers’ system and their computers are impeccably protected, so that that information, protected by GDPR, will not be hacked or misused? Will reviewers even want to share that information?
How should editors inquire about it? And how should they search for reviewers based upon those characteristics (e.g., Publons reviewer database and other bibliographic databases at best contain only affiliation information)? Should diversity of reviewers be achieved for each manuscript, or will stakeholders “accept” if they apply overall, e.g., summary indicators for all manuscripts submitted in 1 year, or over journals lifetime? How will editors balance between required expertise and set goals? If a public diversity database of journals’ identity indicators is made one day (akin to Platform for Responsible Editorial Policies, or SHERPA/ROMEO for preprint policies), what will be counted? For example, if an editor invites 2 men and 2 women to review, and only 2 men accept – will the initial intention to cover sex equity be valued? Will editors manipulate this – and invite those they know will reject the invites? Or even go so far to have agreements with them that they reject reviews? Previous cases have shown cases of suspicious self-citation or citation stacking patterns of journals – how do we ensure this does not happen to diversity indicators? Should journals have mechanisms to enable authors to express preferences or raise concerns about diversity of their reviewers? And if disagreements about them occur, how should they be resolved and by whom? Recommendations and updates from the Joint commitment for action on inclusion and diversity in publishing might soon help answer some of these questions.
Should articles have diversity statements?
Should journals expand declaration sections so that alongside data sharing statements, ethics approvals, conflicts of interests, etc., they also have a diversity section – which would be filled out by authors during submission – and in which they would declare if they considered diversity when they were conducting their study and in analysing their data? Or if they followed Sex and Gender Equity in Research – SAGER guidelines, or Guidance on the Reporting of Race and Ethnicity in Medical and Science Journals? Should a lack of such considerations warrant study rejections? Can editors and reviewers help those questions be considered during study planning, at least for study protocols or registered reports submitted to their journals (see for example a proposed checklist researchers could use for sex-sensitive and gender-sensitive research)? Journals might also consider these topics as a criterion in review and they may make them mandatory in submission templates (as some journals and funders already do). Should a diversity section, also include a statement from journals and reviewers, that diversity aspects have been (appropriately) reviewed? What kind of training will editors and reviewers need to be confident to state this (openly)?
How to decide when and with which identity initiatives to start with?
We are still lacking studies that show effectiveness of intervention to improve the quality of peer review, or even those that say how we should measure the quality of peer review.
Editors might be waiting until identity and diversity studies of excellent quality and generalizability are published, before they start implementing or deprioritize implementing identity policies and practices. For example, a recent study on practices of 145 journals, has indicated that “peer review and editorial processes do not penalize manuscripts by women”. Can editors trust that these findings apply to their own journals and not prioritize asking or predicting/determining the gender of their authors, but rather focus on achieving gender equity of peer reviewers? Do editors owe it to the authors and reviewers to conduct such studies in their own journals? Or to find resources or make public calls for independent researchers to confirm those findings? Which actions should editors implement first? Should they also wait for studies which show which interventions work? We are still lacking studies that show effectiveness of intervention to improve the quality of peer review, or even those that say how we should measure the quality of peer review.
RIPR plans and call for studies
Questions can be a start. Our plans as a journal for the next year are to pilot structured peer review (i.e. a mandatory set of questions each peer reviewer needs to answer, e.g. Do additional study limitations need to be listed?), and in two years pilot publishing summary of changes that occurred due to peer review for each article. However, this year’s Peer Review Week’s focus on identity has encouraged us to do more, and we will start by considering the best way to include diversity considerations in the structured review template, and consulting our advisory board on their view on diversity and gender equity goals. Then comes the survey to authors and reviewers. We also plan to run a pilot on comparing reviews of those we invited as editors, and those who respond to a public call for review of a submitted manuscript shared (through Springer Natures In Review platform or any preprint server). This can perhaps lead to including or volunteering of those who might feel their voices are not being heard.
Finally, please consider submitting any research on peer review diversity to our journal. So much of our work as editors is voluntary, and we lack resources to do all we would want – so please share with us your experiences and interventions you found were (easily) implemented or that helped you achieve your identity, diversity and inclusion goals. And if you are interested in running analyses on our or many other journals that use open peer review, please be in touch via the ‘Contact Us’ link on Editorial Manager.
Disclaimer: I would like to thank colleagues and friends who commented and improved my initial draft considerably, including (in first-name alphabetical order): Ana Jerončić, Anton Ninkov, Bahar Mehmani, Gerben ter Riet, IJsbrand Jan Aalbersberg, Jadranka Stojanovski, Lazaros Belbasis, Lex Bouter, Peter Steenbergen
Comments