Why Peer-Review Is Good for Research and the Experts Who Vet It

For prestigious peer-reviewed publications like the Journal of Advertising Research, dedicated reviewers are high in demand. Indeed, the numbers of contributions often outnumber the available experts to review them.

Although reviewing often is viewed as a chore and time consuming, there are benefits that go beyond simply providing needed service to the profession. Reviewers specify their own areas of expertise and research interest, meaning they likely will receive papers topics that are relevant to their own research. Not only are such papers interesting, reading their literature reviews and seeing their methods helps keep us up to date, and sometimes, changes our thinking.

Reviewing a paper through to publication is valuable for learning how to address and respond to reviewer questions and concerns. I appreciate that JAR, like many journals, sends reviewers a copy of the Editor’s decision along with reviews from the entire review team. Reading through the reviews from other experts, especially as a young faculty member, is an important learning tool. I can see where my comments aligned— or sometimes didn’t—with those of others, enabling me to “calibrate” and develop better judgment moving forward.

There are a lot of different ways to review a paper, but reviewers generally agree on the core issues and concerns: alignment with the journal’s mission, contribution to research, rigor and quality of methods, and novelty of ideas.

Mission

My first concern when reviewing a paper is to make sure it fits with that journal’s stated mission. Published quarterly by WARC for the Advertising Research Foundation (ARF), the JAR is the premier advertising journal for both academics and practitioners. First published in 1960, its stated mission is “to act as the research and development vehicle for professionals in all areas of marketing, including media, research, advertising, and communications.” Moreover, the JAR offers a “forum for sharing findings, applications, new technologies, and methodologies, and avenues of solution.”¹ Ideally, a JAR article marries academic rigor and theory development with clear and actionable practitioner insights. Although an individual paper may tilt more in one direction than the other, I would struggle to support a purely market research-focused or theory-driven paper.

Contribution to Research

Assuming a paper fits the JAR’s mission, I next try to assess its contribution. By contribution, I mean the value a paper provides to the JAR’s audience. Stronger papers signal this early through the title, abstract, and opening paragraphs. The importance of “contribution” also is borne out by a 2016 study about rejected and accepted papers in JAR (Robson, Pitt and West, 2016) co-authored by then editor of the Journal, Douglas C. West. By far the most important factor determining a paper’s eventual acceptance by JAR was its contribution.

By the end of the introduction a reader should have a clear sense of what has been done before in an area, what this particular paper adds, and why this is interesting (See Creswell, 2014; and Grant and Pollock, 2011 for advice on crafting effective introductions). Successful papers either develop new knowledge on an important topic or phenomenon that hasn’t been explored, or they extend existing understanding in creative and valuable ways. Key questions I often ask are

• “Could this paper’s results simply be inferred or intuited from existing theory and research?”
• “Are results surprising or counterintuitive given what we know?” and
• “To what extent does a paper open up a new conversation?”

Similarly, from a practitioner perspective, I wonder

• “To what extent might this paper’s findings change how industry operates?” and
• “Would an MBA class, or a group of executives find this paper interesting?”

Industry publications and the myriad changes brought about by the Internet, mobile devices, and social media, often are good starting points for developing interesting questions. These can then be compared against existing knowledge to distill research ideas that are interesting and to develop new theoretical understanding (See Jaccard and Jacoby 2010 for advice on generating novel ideas).

Rigor and Methods Quality

Whether confidence can be placed in a paper’s findings is my next concern. This gets at the quality of a paper’s design, methods, and statistics. A paper’s design should reflect an author’s research objectives and the current state of the literature. More mature topics tend to gravitate toward theory testing and quantitative studies, while emerging areas tend to be more conceptual, exploratory, and qualitative in nature. Methods should be clearly explicated, with full reporting of any stimuli, measurement items or interview questions, samples, and statistical results.

Authors should note that JAR generally will not publish articles wholly reliant on student- or mTurk samples. They should bear in mind that other researchers should be able to replicate their work based on their reporting. Although such reporting requirements might seem excessive, it is important for a reviewer to fully understand how a study was conducted in order to assess the presence of concerns, such as ordering effects or confounds. Such information can always be trimmed or moved to an appendix at a later point. And, although inference is reasonable, a paper’s conclusions should not overreach its statistical findings.

Finally, I am conscious that all research studies are flawed to some degree (McGrath 1981). Asking “How else could this study have been conducted?” often makes this more apparent. It is only through the larger conversation, of which a paper is only one voice, that confidence in knowledge is built. A paper, therefore, need not be the definitive statement on a topic, but should at minimum carry the conversation forward in a meaningful way. I am also aware that JAR has a relatively short word limit. Although this undoubtedly makes its articles more readable, it also necessarily moderates what can be expected of a paper as a reviewer.

Novelty of Ideas

I described the importance of a paper’s contribution and novelty before discussing methods, for good reason. It is much more difficult—and thereby more valuable—to come up with an interesting idea than it is to execute a method well. This is why I am more favorable to papers that are novel and creative, that make me ask, “Now why didn't I think of that?” Studies can always be redesigned, and statistics can always be rerun if an underlying idea has merit; coming up with a great idea is far more of a challenge.

Raising the Bar

I also try not to forget that my role as a reviewer is not to solely critique, but also to elevate and improve each paper. In some cases, this can involve extensive suggestions on adding or redesigning studies to get at an effect, revising hypotheses to be clearer, adding new moderators or mediators, trying a more sensitive statistical approach, or repositioning a paper’s literature review to enhance its contribution. In others, a reviewer may offer encouragement, suggest potential avenues that might be more fruitful to explore in a new iteration, or point to specific resources helpful in developing a stronger design or more interesting idea for another paper.

Whether or not the paper is accepted, I want the author to feel that the review process was beneficial. I hope that my reviews help authors to improve their research and to write better, and not just different, papers.

Colin Campbell is the 2017 winner of JAR Best Reviewer. He has been on the JAR Editorial Board since 2011. Campbell was Assistant Professor of Marketing at Kent State University when named Best Reviewer. He is now Assistant Professor of Marketing at the University of San Diego.

colincampbell@sandiego.edu

References

Creswell, J. W. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks, CA: Sage, 2014.

Grant, A. M. and T. G. Pollock. “Publishing in AMJ – Part 3: Setting the Hook.” Academy of Management Journal 54, 5 (2014): 873–879.

Jaccard, J. and J. Jacoby. “Chapter 4: Creativity and the Generation of Ideas.” in Theory Construction and Model-Building Skills, New York, NY: Guildford Press, p. 39–74, 2010.

McGrath, J. E. “Dilemmatics: The Study of Research Choices and Dilemmas.” American Behavioral Scientist 25, 2 (1981): 179–210.

Robson, K. E., L. F. Pitt and D. C. West. “Navigating the Peer-Review Process: Reviewers' Suggestions for a Manuscript: Factors Considered before a Paper Is Accepted or Rejected for the Journal of Advertising Research.” Journal of Advertising Research, 55, 1 (2016): 9–17.