Syd Uni sports clinic
06-04-2006, 01:26 PM
Good article on the journal review process from the editor of AJSM:
Jury Selection
Bruce Reider, MD
Chicago, Illinois
"Better the devil you know than the devil you don’t know."
—Irish proverb, collected 1539 by R. Taverner
When The American Journal of Sports Medicine switched to electronic manuscript submission in December 2002, I decided to give authors the opportunity to list individuals whom they thought would be good choices to review their work. I also offered authors the option of identifying anyone whom they would like to exclude as a possible reviewer. Although AJSM cannot accept most of the articles that are submitted for consideration, I want our contributors to feel that their work has been reviewed with erudition and fairness. Giving authors input into the review process is one way to achieve this goal.
When authors do make suggestions, I try to include at least one of their nominees among the reviewers. This is sometimes not possible because the recommended individuals are currently reviewing another manuscript, have just completed a review, or decline to participate. My motivations for following the authors’ suggestions are not completely altruistic, as doing so has brought new experts into our panel of available reviewers. Although the supply of appropriate reviewers in certain subjects can be limited, I have also made it my policy not to assign a paper to anyone excluded by the submitting authors.
Somewhat to my surprise, only about 30% of authors routinely avail themselves of these options. There could be a number of reasons for this, but I like to think that it means that authors are confident that editor-assigned reviewers will be fair and knowledgeable. The peer review process is currently the focus of considerable academic interest and is itself the subject of numerous ongoing research studies.
One such study, reported recently by Schroter et al1 in the Journal of the American Medical Association, compared reviews generated by author-suggested experts to those written by editor-suggested ones. The study examined 10 journals that span a variety of medical specialties and that routinely allow their contributors to suggest or exclude reviewers. First, the reviews were blindly evaluated for quality by trained raters according to a validated rating system.2 A further analysis was carried out within journals whose reviewers are asked to make recommendations regarding acceptance or rejection. Among these, papers with discordant reviews were scrutinized to determine whether the editor- or author-suggested reviewers were more likely to generate a favorable recommendation. Finally, the study examined whether, when the publication recommendations differed between reviewers, the editor was more likely to agree with the author- or editor-suggested reviewer.
The results of the study were intriguing. The quality ratings of the reviews generated by author- and editor-suggested reviewers were nearly identical. The author-suggested reviewers were 1.64 times (95% confidence interval, 1.02–2.66) more likely than the editor-suggested reviewers to recommend acceptance compared to revision or rejection. This tendency was even more marked in the one journal that routinely revealed the reviewers’ identities to the submitting authors. For this journal, the author-suggested reviewers were 12.4 times (95% confidence interval, 1.60–95.8) more likely to recommend acceptance. When the reviewers in the study of Schroter et al1 differed in their publication recommendations, the author-suggested reviewers made the more favorable recommendation 70.8% of the time. In the final manuscript decision, the editor sided with the author-suggested reviewers about half the time and the editor-suggested ones the other half.
One might speculate that authors recommend reviewers whom they think (or know) might be inclined to generate favorable reviews. This is probably at least occasionally true because some AJSM contributors have suggested reviewers from their own institutions. If an author feels that a specific expert is likely to produce a low quality or unfair review, it seems logical that he or she would not recommend that person as a reviewer or would even ask to have that person excluded. In their article, Schroter et al1 concluded that the most plausible explanation for the tendency of author-recommended reviewers to produce more favorable recommendations was that authors nominate knowledgeable experts in their field of research who would be likely to recognize the importance of their work. Indeed, this interpretation is supported by the finding that in the case of discordant recommendations, the editors were as likely to agree with the author-suggested reviewers as with the ones they had themselves selected. This in turn suggests that editors pay more attention to the content and quality of reviews than they do to whether the reviewers were nominated by the authors or themselves. This is indeed the case at the AJSM.
In the interest of contributor empowerment, I encourage authors submitting papers to the AJSM to remember the reviewer recommendation/exclusion options. I do urge them not to confine their recommendations to members of the editorial board, who are already heavily used. It is a benefit to both the Journal and its readers to have a selection process that is as fair and knowledgeable as possible.
REFERENCES
Schroter S, Tite L, Hutchings A, Black N. Differences in review quality and recommendations for publication between peer reviewers suggested by authors and by editors. JAMA. 2006;295:314–317.[Abstract/Free Full Text]
van Rooyan S, Black N, Goodlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol. 1999;52:625–629.[CrossRef][ISI][Medline][Order article via Infotrieve]
Jury Selection
Bruce Reider, MD
Chicago, Illinois
"Better the devil you know than the devil you don’t know."
—Irish proverb, collected 1539 by R. Taverner
When The American Journal of Sports Medicine switched to electronic manuscript submission in December 2002, I decided to give authors the opportunity to list individuals whom they thought would be good choices to review their work. I also offered authors the option of identifying anyone whom they would like to exclude as a possible reviewer. Although AJSM cannot accept most of the articles that are submitted for consideration, I want our contributors to feel that their work has been reviewed with erudition and fairness. Giving authors input into the review process is one way to achieve this goal.
When authors do make suggestions, I try to include at least one of their nominees among the reviewers. This is sometimes not possible because the recommended individuals are currently reviewing another manuscript, have just completed a review, or decline to participate. My motivations for following the authors’ suggestions are not completely altruistic, as doing so has brought new experts into our panel of available reviewers. Although the supply of appropriate reviewers in certain subjects can be limited, I have also made it my policy not to assign a paper to anyone excluded by the submitting authors.
Somewhat to my surprise, only about 30% of authors routinely avail themselves of these options. There could be a number of reasons for this, but I like to think that it means that authors are confident that editor-assigned reviewers will be fair and knowledgeable. The peer review process is currently the focus of considerable academic interest and is itself the subject of numerous ongoing research studies.
One such study, reported recently by Schroter et al1 in the Journal of the American Medical Association, compared reviews generated by author-suggested experts to those written by editor-suggested ones. The study examined 10 journals that span a variety of medical specialties and that routinely allow their contributors to suggest or exclude reviewers. First, the reviews were blindly evaluated for quality by trained raters according to a validated rating system.2 A further analysis was carried out within journals whose reviewers are asked to make recommendations regarding acceptance or rejection. Among these, papers with discordant reviews were scrutinized to determine whether the editor- or author-suggested reviewers were more likely to generate a favorable recommendation. Finally, the study examined whether, when the publication recommendations differed between reviewers, the editor was more likely to agree with the author- or editor-suggested reviewer.
The results of the study were intriguing. The quality ratings of the reviews generated by author- and editor-suggested reviewers were nearly identical. The author-suggested reviewers were 1.64 times (95% confidence interval, 1.02–2.66) more likely than the editor-suggested reviewers to recommend acceptance compared to revision or rejection. This tendency was even more marked in the one journal that routinely revealed the reviewers’ identities to the submitting authors. For this journal, the author-suggested reviewers were 12.4 times (95% confidence interval, 1.60–95.8) more likely to recommend acceptance. When the reviewers in the study of Schroter et al1 differed in their publication recommendations, the author-suggested reviewers made the more favorable recommendation 70.8% of the time. In the final manuscript decision, the editor sided with the author-suggested reviewers about half the time and the editor-suggested ones the other half.
One might speculate that authors recommend reviewers whom they think (or know) might be inclined to generate favorable reviews. This is probably at least occasionally true because some AJSM contributors have suggested reviewers from their own institutions. If an author feels that a specific expert is likely to produce a low quality or unfair review, it seems logical that he or she would not recommend that person as a reviewer or would even ask to have that person excluded. In their article, Schroter et al1 concluded that the most plausible explanation for the tendency of author-recommended reviewers to produce more favorable recommendations was that authors nominate knowledgeable experts in their field of research who would be likely to recognize the importance of their work. Indeed, this interpretation is supported by the finding that in the case of discordant recommendations, the editors were as likely to agree with the author-suggested reviewers as with the ones they had themselves selected. This in turn suggests that editors pay more attention to the content and quality of reviews than they do to whether the reviewers were nominated by the authors or themselves. This is indeed the case at the AJSM.
In the interest of contributor empowerment, I encourage authors submitting papers to the AJSM to remember the reviewer recommendation/exclusion options. I do urge them not to confine their recommendations to members of the editorial board, who are already heavily used. It is a benefit to both the Journal and its readers to have a selection process that is as fair and knowledgeable as possible.
REFERENCES
Schroter S, Tite L, Hutchings A, Black N. Differences in review quality and recommendations for publication between peer reviewers suggested by authors and by editors. JAMA. 2006;295:314–317.[Abstract/Free Full Text]
van Rooyan S, Black N, Goodlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol. 1999;52:625–629.[CrossRef][ISI][Medline][Order article via Infotrieve]