ethics

Transformation and Promoting Trust in the Peer Review Process

Posted by

0

Today is the start of Peer Review Week, an annual global event celebrating the essential role that peer review plays in maintaining scientific quality. This year’s focus is on trust in peer review, and this post addresses the evolving transformation of the peer review in scientific publication.

Peer review continues to develop, albeit slowly, in terms of models and methods, with increasing calls for openness and transparency. There are 3 common forms of peer review:

  • Double-blind review: Authors’ and reviewers’ identities are hidden from each other in an attempt to minimize bias.
  • Single-blind review: Authors identities are revealed to all, but reviewer identities are not revealed to authors (also known as anonymous review)
  • Open review: Author and reviewers are identified are revealed and various levels of the process and outputs may or may not be made public

Types of open review, with increasing levels of openness, include the following:

  • Level 1: Reviewer and author identities are revealed to each other during the peer review process
  • Level 2: Indication of editor and/or reviewer names on the article
  • Level 3: Posting of peer review comments with the article, signed or anonymous
  • Level 4: Publication of peer review comments (signed or anonymous) with authors’ and editors’ responses, decision letters, and submitted and revised manuscripts
  • Level 5: Publication of the submitted manuscript after a quality check and inviting public discussion from the community

A recent look at the types of peer review used by some top-ranked general medical and science journals shows that most journals use single-blind review, with some allowing reviewers to choose to sign their reviews. For example, JAMA has a single-blind review process and offers reviewers the option to sign their reviews that are shared with authors, and copies of reviews are shared with other reviewers.

JAMA also has an editorial collaborative process, called editorial review before revision, during which senior editors, a manuscript editor, and an editor with expertise in data display collaborate to provide guidance to the authors on all that is needed during revision to reach a favorable final decision.

However, these processes are not public. A short video that explains an inside view of the editorial and peer review process at JAMA is available.

JAMA Network

The BMJ has a fully open review process with the following published with all research articles: all versions of the manuscript, the report from the  manuscript committee meeting, reviewers’ signed comments, and authors’ responses to all comments from editors and reviewers. Nature publishes reviewer names and comments and author rebuttal letters; however, authors and reviewers can opt out of the open review process. And eLife has a mixed model with reviewers’ names revealed to each other during the review process; decision letters, anonymous reviewer comments, and author response letters published with the article; and an option for reviewers to sign their reviews.

One of the earliest demonstrations of open and collaborative peer review was launched in 2001 by Copernicus Publications, an open-access publisher of scientific journals. These journals use a 2-stage process:

“In the first stage, manuscripts that pass a streamlined access review are immediately posted as preprint in the respective discussion forum. They then undergo an interactive public discussion, during which the referees’ comments (anonymous or attributed), additional community comments by other members of the scientific community (attributed), and the authors’ replies are posted. In the second stage, the peer-review process is completed and, if accepted, the final revised papers are published in the journal.”1

Many studies have compared the quality of single-blind, double-blind, and open review. Early randomized trials2,3 found no differences in the quality of double-blind, single-blind, or open review. But some studies have found differences, such has higher quality for blinded review,4 higher quality for signed reviews,5 and higher quality for open review.6 And some studies7,8 have identified biases that may be better managed with double-blind review (eg, bias toward gender, geography, institutions, and celebrity authors).  However, no study has yet compared the quality of published articles that have undergone these different types of peer review.

Drummond Rennie, the founder of the International Congress on Peer Review and Scientific Publication, has been a vocal proponent of open peer review. Writing about freedom and responsibility in publication in 1998, Rennie commented,

“The predominant system of editorial review, where the names of the reviewers are unknown to the authors, is a perfect example of privilege and power (that of the reviewer over the fate of the author’s manuscript) being dislocated from accountability….to the fellow scientist who wrote the manuscript. For that reason alone, we must change our practices. ….The arguments for open peer review are both ethical and practical, and they are overwhelming.”9

There have also been numerous studies demonstrating the feasibility of each type of peer review. However, some studies have found that double-blind review is not always successful and have reported rates of failure to ensure blinding ranging from 10% to 40%. Other studies have found that reviewers who are asked to sign their reviews may be more courteous or positive in their recommendation, may take longer to complete their reviews, and may be more likely to decline invitations to review.

Support for open review, with options, continues to evolve. In a 2016 OpenAire survey of 3062 academic editors, publishers, and authors,10 60% indicated that open peer review (“including making reviewer and author identities open, publishing review reports and enabling greater participation in the peer review process”) should be common in scholarly practice, but they had some concerns. For example, 74% responded that reviewers should be able to choose to participate in open review, and 67% reported being less likely to review if open review was required.

The Nature journals have been experimenting with various models of peer review, and in 2016, Nature Communications announced that about 60% of its authors agreed to have their reviews published.11  In 2019 and 2020, Nature journals began offering “transparent peer review” with options for authors and reviewers to opt out.12

Elsevier conducted a pilot of open review from 2014 to 2017 in 5 journals, with reviews published.13 During this pilot, younger and nonacademic scholars were more willing to review and provided more positive and objective recommendations. There was no change in reviewer willingness to review, their recommendations, or turn-around times. But, only 8% of reviewers agreed to reveal their identities with the published reviews.

Thus, the key to successful transformation to open peer review and maintaining trust in the process may be offering options to authors and reviewers. Whichever model is used, journals should clearly and publicly describe their processes (eg, in Instructions for Authors) and continue to evaluate and test ways to improve the peer review process for authors, reviewers, and editors.–Annette Flanagin, Executive Managing Editor and Vice President, Editorial Operations, for JAMA and the JAMA Network, and Executive Director of the International Congress on Peer Review and Scientific Publication

*Note: Portions of this post have been presented at several meetings.

References:

  1. Copernicus Publications. Interactive peer review. Accessed August 23, 2020. https://publications.copernicus.org/services/public_peer_review.html
  2. Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D; PEER Investigators. Does masking author identity improve peer review quality? a randomized controlled trial. JAMA. 1998;280(3):240–242. doi:10.1001/jama.280.3.240 https://jamanetwork.com/journals/jama/fullarticle/187758
  3. van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA. 1998;280(3):234–237. doi:10.1001/jama.280.3.234 https://jamanetwork.com/journals/jama/fullarticle/187750
  4. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review: a randomized trial. JAMA. 1990;263(10):1371–1376. doi:10.1001/jama.1990.03440100079012 https://jamanetwork.com/journals/jama/fullarticle/380957
  5. Walsh E, Rooney M, Appleby L, Wilkinson G. Open peer review: a randomised controlled trial. Br J Psychiatry. 2000;176(1):47-51. doi:10.1192/bjp.176.1.47
  6. Bruce R, Chauvin A, Trinquart L, et al. Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis. BMC Medicine. 2016;14(85). https://doi.org/10.1186/s12916-016-0631-5
  7. Lerback J, Hanson B. Journals invite too few women to referee. Nature. 2017;541(7638):455–457. doi:10.1038/541455a
  8. McGillivray B, De Ranieri E. Uptake and outcome of manuscripts in Nature journals by review model and author characteristics. Res Integr Peer Rev. 2018; 3(5). https://doi.org/10.1186/s41073-018-0049-z
  9. Rennie D. Freedom and responsibility in medical publication: setting the balance right. JAMA. 1998;280(3):300–302. doi:10.1001/jama.280.3.300
  10. Ross-Hellauer T, Deppe A, Schmidt B. Survey on open peer review: attitudes and experience amongst editors, authors and reviewers. PLOS One. 2017;12(12): e0189311. https://doi.org/10.1371/journal.pone.0189311
  11. Transparent peer review one year on. Nat Commun. 2016; 7(13626). https://doi.org/10.1038/ncomms13626
  12. Peer review policy. Nature Journals. Accessed August 23, 2020. https://www.nature.com/nature-research/editorial-policies/peer-review#transparent-peer-review
  13. Bravo G, Grimaldo F, López-Iñesta E, et al. The effect of publishing peer review reports on referee behavior in five scholarly journals. Nat Commun. Published online Junary 18, 2019. https://doi.org/10.1038/s41467-018-08250-2

How Many Is They?

Posted by

1

Since I’ve been a manuscript editor, JAMA Network journals have published a few articles about health care for transgender patients. I’ve had the good luck to edit a few—they are always interesting—but this week, I realized that there is a grammatical issue in editing these articles that I have never heard fully addressed.

The issue is not what pronouns to use for transgender individuals—that question is well-known. Because the English language uses gendered pronouns, people who change their gender expression or whose gender isn’t accurately defined by labels are faced with several choices: should they go by she, he, a singular they, or a neologism, such as xe?

The news media has addressed this, including The New York Times articles in 2016 and 2017, with another written by transgender English professor Jennifer Finney Boylan in 2018. These articles often make the same points: that people can get confused by this transformation of language, but that people who want to use pronouns that reflect a gender different from their assigned sex should have their wishes respected. This squares with the approach used in JAMA Network journals.

But it also raises a question rarely addressed: what about verb conjugation? In the present tense, English applies a letter s to the third-person singular (he, she, or it runs) but not to the first-person singular, first-person plural, second-person singular, and third-person plural (I, we, you, and they run). So, if an article uses a singular they, should it be conjugated like the third-person plural (run) or like the third-person singular (runs)?

While editing, I have realized that the default method of using the singular they along with someone’s name (or a descriptor, such as the patient) involves flip-flopping between singular and plural verb conjugations (eg, “The patient is receiving gender-affirming treatment, and they are pleased with the outcome thus far”). A consistent use of the singular they would seem to require using third-person singular conjugations throughout (“The patient is receiving treatment…. They is pleased…”) or plural conjugations throughout (“The patient are receiving treatment…. They are pleased….”). Is either approach correct?

The public discourse on pronouns has not provided much insight. It has prompted the argument that we all use forms of singular they-series pronouns in casual language, in sentences such as “If someone wants gender-affirming treatment, that is their choice” or “Give them an injection.”

But that is distinct from the current issue. In these usages, the pronoun is usually objective (them) or possessive (their or theirs), and the verb is conjugated in the third-person singular alongside another subject. (In the example sentences, these are “someone” and “that,” and in the command, an unspoken “you.”) When the singular they is placed immediately before the verb, the question of conjugation instantly reappears.

Weighing “they run” vs “they runs,” I looked for guidance in the AMA Manual of Style. It offered a few relevant thoughts: “In an effort to avoid both sex-specific pronouns and awkward sentence structure, some writers use plural pronouns with singular indefinite antecedents… Editors of JAMA and the Archives Journals prefer that agreement in number be maintained in formal scientific writing.”

But that seemed short of clear instructions. Still uncertain if constructions like “they is pleased” would please anyone, I reached out to Jennifer Finney Boylan, the New York Times contributor whose article on pronouns was published in 2018.

A quick email to her university address got an equally quick response. “I’m not certain about this,” she wrote. “I want to endorse ‘they is’ because the non-specific pronoun is still referring to a singular individual. On the other hand, ‘they are’ sounds better to my old, English professor ears.”

That seemed similar to how our society is handling the matter. On her authority, I decided to stick with conjugations that shift from singular to plural and let the matter rest until a thoughtful body of grammarians, popular opinion, and perhaps the next edition of the New York Times’ article series on pronouns weigh in with a definitive answer to conjugating verbs after a singular they.—M. Sophia Newman

Patient Privacy

Posted by

0

Sometimes before I go to bed, I like to check in on one of my favorite YouTubers, Dr Pimple Popper (the nom de internet of dermatologist Sandra Lee), who posts videos of dermatologic procedures and skin care treatments. I particularly enjoy watching videos of dilated pore extractions, and I don’t mind watching lipoma extractions either (although I do sometimes fast-forward through the excisions). I know these types of videos can get viewers’ stomachs churning a bit, but I think it’s no worse than various photographs in medical journals I have worked at over the years. And because of my occupation, I do wonder about patient privacy and anonymity.

Patients featured on this YouTube channel may have a cyst near their eye or ask for blackheads to be removed from their cheek, and their faces are clearly visible. In many videos, Dr Lee chats with her patients, and although she sometimes edits out personal details, some of it stays. Dr Lee says that patients do sign consent forms before videos are published.

Similarly, when manuscript editors of medical journals encounter photographs of patients, we must review whether the photograph might intrude on patient privacy. Authors must obtain written permission from patients (or their legally authorized representatives) for any descriptions, photographs, or videos of patients or identifiable body parts and indicate that such consent was obtained in the Methods or Acknowledgment section. When I started in this field as an editorial assistant, I processed a manuscript that described a skin lesion on a patient’s back. In an accompanying photograph, the patient’s distinctive tattoo was visible, and I needed to ask the author to either obtain patient consent or have the photograph cropped because the patient (as well as anyone who knew he had that tattoo) would be able to identify himself. Results of imaging studies and photos of laboratory slides may also have identifying information that should be removed.

Protecting patient privacy also extends to what is in the text of an article. When editing case descriptions, case reports, and personal essays, nonessential identifying data (eg, sex, specific ages, race/ethnicity, occupation) should generally be removed unless the author has permission or the information is clinically or scientifically relevant and important. Authors and editors should not falsify or fictionalize details; doing so may introduce false or inaccurate data.

Read more about patient’s rights to privacy and anonymity in section 5.8.2 of the AMA Manual of Style.—Iris Y. Lo

Have You Talked to Your Tables About the Dangers of Sex Bias?

Posted by

0

The problem of bias is well documented in the biosciences. Even since the Health Revitalization Act of 1993, which laid out guidelines intended to ensure more equitable representation of women and minorities in federally funded scientific research, the problem persists. A 2010 study published in The Journal of Women’s Health found that, among 46 clinical studies enrolling both sexes, women comprised on average 37% of the participants, and among 69 studies, 87% did not conduct analyses by race or ethnicity, and 18% did not report differences in the racial makeup of the study sample at all. Examples of this sort abound and, setting aside the pernicious sociohistorical and nuanced biologic reasons for this phenomenon, the resulting reality is that medicine, as applied to women and minorities, is less evidence based because most research is extrapolated from a homogeneous population—white men.

But even as we attempt to resolve these problems—ensuring that guidelines are in place and that they are followed when conducting new research—there is another, more subtle way that these biases creep into the biomedical literature. Even if the study itself was conducted using a diverse population of participants, sometimes the reporting elides this fact.  As a manuscript editor I have encountered this problem more often than one would expect, and the culprit is usually the table.

In this table, as in many tables that I have encountered, “white” and “male” are the default. Women’s bodies and the bodies of racial and ethnic minorities are implied by the number of white male bodies present.

A good rule when presenting data in tables is to make sure that when you are reporting the sex of participants, if you choose to report only 1 sex, choose the sex that constitutes the majority of the sample. When reporting on racial and ethnic differences, be as specific as possible (even if these comprise a small percentage of participants). Who are the “others?”

The current edition of the AMA Manual of Style does not explicitly lay out these precautions, but in chapter 4, section 1, you will notice that every example shown for presenting data in tables follows these guidelines.

This is not merely a problem of “political correctness” or social equity—it is a question of accurate reporting and just plain good science.—Gabriel Dietz

 

 

 

 

Go, Embargo, Go

Posted by

1

So what’s an embargo, anyway? There’s the economic trade kind, but let’s stick to the news kind (much more relevant for AMA Style Insider readers). I spoke with Jann Ingmire, the JAMA and Archives Journals media relations guru, and she explained that embargoes exist primarily to give reporters the opportunity to cover a story in a more thorough way.

Here’s how they work: Embargoed material is released to members of the press prior to being released to the public, usually a few days early. This gives reporters time to do research, conduct interviews, and write a really great piece. When the embargo lifts, journalists are already prepared to report on newly published scientific studies.

Most of the time, the system works, but occasionally, an embargo is broken. Ms. Ingmire said she tries to give reporters the benefit of the doubt because, usually, it’s simple human error. Sometimes, though, the embargo break is flagrant. When this happens, reporters are sanctioned and stop receiving embargoed material.

Embargoes make it possible for everyone—from the independent blogger to the major media outlet—to have the same opportunity to gather a story. If you want to learn more, read embargowatch.wordpress.com, a blog that chronicles how embargoes affect news coverage.—Lauren Fischer

Minority Report

Posted by

0

Risk factors included racial/ethnic minority status, male sex, higher hemoglobin A1c level, use of insulin, longer duration of diabetes, and higher systolic blood pressure.—From This Week in JAMA, August 11, 2010.

The meaning of the adjective minority in this statement may appear to be clear to the reader. However, what constitutes minority status for race/ethnicity? In which country, population, or time? And in this context—risk factors for diabetic retinopathy—why would minority status per se (as opposed to a specific genetic background, or perhaps lower socioeconomic status and resulting lack of access to health care) be a risk factor? This usage points to a common but probably outdated use of the term minority to refer to a population of people. Why is this term to be avoided in this context?

Merriam-Webster’s Collegiate Dictionary1 defines minority as “a part of a population differing from others in some characteristics and often subjected to differential treatment; … a member of a minority group (an effort to hire more minorities)” and The American Heritage Dictionary of the English Language2 states, “A racial, religious, political, national, or other group regarded as different from the larger group.” These definitions evolved from the more basic meaning of minority. Insofar as usage and equity are concerned, however, historian Amoja Three Rivers stated emphatically 20 years ago that “at least four-fifths of the world’s population consists of people of color. Therefore, it is statistically incorrect as well as ethnocentric to refer to [them] as minorities. The term ‘minority’ is used to reinforce the idea of people of color as ‘other.’”3 The white race is becoming a “minority” in many countries where that had previously not been the case, including the United States. Nor are women to be considered minorities simply by their numbers (as an example of this, in the United States in the academic year 2008–2009, among first-year enrollees in medical school, there were 9619 men and 8889 women, compared with 10 576 men and 6205 women in 1988–19894).

When used as a noun to describe and thus label and marginalize a racial/ethnic, gender-specific, physically disabled, or other group of persons with a common trait, minority (and minorities) is an exclusive term that should be avoided.

Note: Minority (and majority) is appropriate to use when describing, for example, the count (number) in an election, an opinion of a nation’s high court, or in common usage to mean less than half (or more than half) of a given sample.—Roxanne K. Young, ELS, with thanks to Margaret A. Winker, MD

1. Merriam-Webster’s Collegiate Dictionary. 11th ed. Springfield, MA: Merriam-Webster Inc; 2003.

2. The American Heritage Dictionary of the English Language. 3rd ed. Boston, MA: Houghton-Mifflin Co; 1992.

3. Maggio R. Talking About People: A Guide to Fair and Accurate Language. Phoenix, AZ: Oryx Press; 1997:273.

4. Barzansky B, Etzel SI. Medical schools in the United States, 2008–2009. JAMA. 2009;302(12):1353(appendix I, table 2).

Mistakes Were Made

Posted by

0

The January-March 2011 issue of Science Editor is devoted to the theme of correcting the scientific literature, something that happens to our journals on a regular basis. In 2011 alone, we’ve already logged more than 70 corrections. With the advent of rushing to get everything online as soon as possible, sometimes it seems like we are doing more corrections than ever. Or maybe it just feels that way because with online articles, we can go in and fix that darn typo rather than just let it be. Hard to say.

You can check out the post about it at the Science Editor Editor’s Blog, and (if you are a Council of Science Editors member) read the whole thing at the CSE site. Also, feel free to leave a comment and let us know how your publication handles corrections in print and online.—Lauren Fischer

Who Was That Masked Manual?

Posted by

0

No one wants to be a don’t, a fashion faux pas splashed in full color across the page of a widely circulated fashion magazine. In an attempt to protect the identity of the don’t, one such periodical places black bars over the eyes of each fashion offender. Is this technique sufficient to spare the person the potential humiliation and ridicule that may follow? The AMA Manual of Style offers a resounding no.

Reports of medical cases and research can be greatly enhanced by a photographic or illustrative example of the condition under discussion. Imagine reading a case report of a unique rash with no accompanying photograph, or an article about a newly developed surgical technique without before-and-after images or an illustration of the surgeon’s approach. A picture, it is often said, is worth a thousand words (see 7.5.4, Clichés).

Medical journals have a responsibility to protect patients’ rights to privacy and anonymity, including the deletion of patients’ names, initials, and assigned numbers from case reports; the removal of identifying information from radiographs, digital images, and laboratory slides; and the deletion of identifying details from descriptions of patients or study participants in published articles (see 5.8.2, Patients’ Rights to Privacy and Anonymity).

Until the late 1980s, placing black bars over the eyes of patients in photographs was accepted as a way to protect their identities. However, some journals began to discontinue this practice when it became apparent that bars across the eyes do not preserve confidentiality (see 5.8.2, Patients’ Rights to Privacy and Anonymity). The Uniform Requirements for Manuscripts Submitted to Biomedical Journals concurs, noting that “masking the eye region in photographs of patients is inadequate protection of anonymity.”1

When detailed descriptions, photographs, or videos of faces or identifiable body parts are included with a manuscript that might permit any patient to be identified (even if only by the patient himself or herself), authors should obtain written permission from the identifiable patients (or legally authorized representatives) to publish the information. An example of the patient permission form used by JAMA and the Archives Journals is available at http://pubs.ama-assn.org/misc/patientconsent.pdf. The journal may wish to state (eg, in the Acknowledgment) that the patient shown and/or described has provided written permission.

If such permission cannot be obtained, another approach the journal might take is to remove nonessential potentially identifying details. For example, a patient described as a “20-year-old woman from suburban St Louis with end-stage renal disease undergoing dialysis” could be rewritten as a “young adult woman with end-stage renal disease undergoing dialysis.”

However, altering descriptive characteristics (eg, sex) is not appropriate because it is a form of falsification and may be misunderstood by readers and others conducting secondary analyses of published reports. The best way to provide complete information while protecting patient rights is to obtain informed consent for publication. Informed consent for this purpose requires that an identifiable patient be shown the manuscript to be published or waive his/her right to review the paper.1

To protect patients and to spare potential legal woes, medical journals should not use black bars on photographs in the attempt to preserve confidentiality. The image should be replaced with one that has permission for publication or one in which identifiability is not an issue, or the article should run sans photograph.—Stacy L. Christiansen, MA

1. Uniform Requirements for Manuscripts Submitted to Biomedical Journals: ethical considerations in the conduct and reporting of research: privacy and confidentiality—patients and study participants. http://www.icmje.org/ethical_5privacy.html. Accessed May 9, 2011.