This is a post on how to rebut reviews. Specifically, CHI reviews. The ideas presented here probably apply to rebuttals more broadly, but much of what I’m going to speak about in this post is specific to CHI.
Okay, so the long, long — so long — wait is over and you have your CHI reviews! Now, you’re wondering if you should rebut at all and, if you do, how you should go about it.
Let’s consider the first point: should you rebut? Good question! Follow this simple flowchart:
- If your scores are high and acceptance is all but a formality (i.e., you have an average score of 4 or above): Congrats! Also, you should write a rebuttal.
- If your scores are middling and acceptance seems possible but not certain (i.e., you have an average score of 3 – 4): Congrats! Also, you should write a rebuttal.
- If your scores are low and acceptance seems like a long shot: I’m sorry to hear that. Also, you should write a rebuttal.
Okay, sorry for being facetious. There may be valid situations where you should not write a rebuttal. For example, your paper is not likely to get in and you have a lot of other things on your plate at that moment. [1] But, if you’re reading this article, you’re probably not in that camp.
Okay, why should you write a rebuttal? Lots of reasons, here’s just a few:
- First, it’s a good forcing function for you to critically reflect on your own work and how it can be improved. When you rebut, you have to articulate why you think your review comments are unjustified / ridiculous / unfair. In the process of doing so, you’ll likely either understand where reviewers are coming from and/or learn how to improve your paper so future reviewers won’t have the same misconceptions.
- Second, it can actually change reviewers’ minds. It’s rare, but it happens. I’ve had several review scores increase by half a point or more through the rebuttal process. You’ve already invested dozens of hours into preparing this submission: Why not spend a few more to give your work the best chance it can get?
- Third, not rebutting can actually reflect poorly on your paper. Ultimately, you want your AC to champion your paper during the PC meeting. A rebuttal is a good way to give your AC talking points for why your paper should be accepted despite its inevitable shortcomings (no paper is perfect, right? Certainly no paper of mine). No rebuttal can be perceived as a lack of caring on your part.
- Finally, it’s a great way to justify to your adviser (or to yourself) why you’ve done no research this week.
Alright, so you should rebut. How? I’m going to share what I’ve learned through six (ugh) years of submitting to CHI as a Ph.D. student in an HCI program, along with 5 years of reviewing for CHI as an external reviewer. Note: I make no promises of this being the best method. But, it has worked for me in the past.
How to Rebut
Step 1: Understand your objective
Your rebuttal IS a means to discuss concrete and realistic changes you can make to improve your paper in light of reviewer feedback. Concrete means specific. If your reviewers have misunderstood something, explain how you will change your document to make things clearer. Realistic means doable within the page limits and within the three to four weeks you have until the camera ready deadline.
Your rebuttal is NOT a cathartic condemnation of your reviewers. [2] Your rebuttal is also NOT a comprehensive FAQ. You do not have to (and probably cannot) address all of the issues raised.
Step 2: Summarize
Okay, now that we’re on the same page, here’s how to start. Ideally, you want to read your reviews and set them aside for at least 24 hours. Knee-jerk responses to reviews are rarely productive. You want some emotional separation between the you-who-read-the-reviews and the you-who-writes-the-rebuttal.
Once your 24 hours have passed, your first task is to distill each review into a set of individual points that would fit on a post-it note.
The points can include strengths as well as weaknesses. Often, one reviewer will point out a strength that another reviewer will consider a weakness. Distilling strengths makes it easier to understand when an issue is simply a subjective disagreement versus when an issue points out something you may need to address.
Tag each point with the reviewer or committee member who brought it up. Often, your primary (or 1AC) will list points that he or she finds especially pertinent in a meta-review. Pay special attention to those.
My list ends up looking something like:
R1: Discussion is vague and fragmented and should tie-in to overarching motivation.
1AC: Unclear on how many training samples needed.
etc…
Step 3: Synthesize
Have you ever affinity diagrammed? Yes? Okay, then you’re going to do that next with all of the non-strength points you listed in Step 2. No? I’m going to teach you, at a very basic level, how to affinity diagram.
Affinity diagramming is just a form of clustering. You group together points that have “affinity” towards each other to make a set of clusters that each capture an idea. So, you go from have a list of vaguely connected points to a smaller list of clusters that all capture a specific, concrete idea.
The best piece of advice I can give you about making these clusters is to make them prescriptive, not descriptive.
Descriptive clusters group together points that describe something similar on the surface. One reviewer might be confused about your motivation, while another might be confused about your study design. On the surface, the two points seem related: they are both misconceptions. But, they do not share a solution (apart from, maybe, better writing).
Prescriptive clusters, on the other hand, group together points that “prescribe” a single solution. For example, if one reviewer is confused about your study design and another wants to more details about participant demographics, then that suggests that your methodology section needs to be clearer.
So, take your list of (non-strength) points in Step 2 and make a set of prescriptive clusters. Then, name those clusters. My clusters end up looking something like:
Contextualize contribution
- 1AC: The paper is disconnected from prior art.
- R3: How is this different from System Y?
Clarify study design
- R5: How many participants have used an Android phone before?
- R2: Was a power analysis conducted?
etc.
Step 4: Triage
Chances are you have far more clusters than you can address in 5000 characters.
So, you need to prioritize. Here’s how.
First, assign two binary ranks to all of your clusters: importance and ease of addressing. By binary I mean high or low. Weight AC comments higher in determining importance. Otherwise, importance can be determined by asking yourself the question: “If this were not an issue, would my score be a lot higher?” It’s hard to answer, I know, but you should have some feel for it based on your own knowledge of the issue and reviewer valence.
Once you’ve ranked your clusters along the dimensions of importance and ease of addressing, you should prioritize your clusters as follows: (1) important and easy to address, (2) easy to address but not important, (3) important but not easy to address and (4) not important and not easy to address.
I know it’s weird to prioritize “easy to address but not important” above “important but not easy to address”, but reviewers are unlikely to believe that you are going to make big changes in a few weeks. Remember, too, that the review process is double blinded, so even if you are a super hard-working hacker who has a track record of Doing Whatever It Takes, reviewers don’t know that. So, it’s best to stick to making a case for what the average CHI submitter can realistically do and change in a few weeks.
Now, you can create an ordered list of points you should address in your rebuttal.
Step 5: Rebut
Finally! Time to write. Use your 5000 characters to address as many as of the prescriptive issue clusters you’ve identified as you can, in their order of priority.
For each cluster, I recommend creating a header with the cluster name along with a parenthetical list of all the reviewers who raised an issue related to that cluster.
Then, if you’re addressing a misunderstanding (e.g., what a figure means), you should clarify the misunderstanding and outline, specifically, what you will change in the manuscript to mitigate the chances that a future reader will have the same misunderstanding.
If you’re addressing a broader issue (e.g., adding related work), you should summarize the changes you plan to make in specific sections of the paper. Your summarization should contain actual content that you plan to add, if necessary (e.g., the actual results of a supplementary analysis you will run). It should also outline what you plan to remove from the paper to make room for the new content, if necessary.
Finally, I personally like to put in estimates for the expected amount of time it will take me to make these changes (e.g., 1 week to do an additional analysis, plus 3 days to write the new results in). This helps me make a rhetorical argument for why I believe the changes can be made within a realistic time-scale.
Continue this process for every issue in your cluster list for which you have space, in order of priority.
Final Deconstruction
At the end of the process, your rebuttal should look something like this:
- One to two sentences of acknowledgment and appreciation.
- A short summarization of issues as you understand them (through the above process).
- List of rebutted issues:
– Header (with shout outs to specific reviewers)
– Summarization of issue
– Strategies to address it
– Expected time / space commitment - Concluding statement (one to two sentences reiterating your appreciation as well as your plan for addressing the issues that were raised).
Oh, one last point: Use white space liberally, even though it eats away at your precious character count. Information presentation is important. Ultimately, you’re hoping to convince your AC to champion your paper in the PC meeting, and AC’s often have to wade through dozens of these rebuttals in a very abbreviated time frame.
Alright, hope that helps. Now go forth and get those points back, you academic ninja, you. [3]
Footnotes
[1] You should still write a quick note thanking your reviewers (even if you really do not want to thank them). You did sort of sign up to receive their opinion on your work, and, even if they were mean about it, they did spend some of their valuable time considering your work.
[2] By the way, rebuttals can also lower your score if you’re not careful.
[3] You may not get any points back. Rebuttals don’t change a lot of minds, but they can occasionally change some minds.
One final note: If you liked this post and would like to show your support, here are two things you can do:
– Follow me on Twitter @scyrusk; and,
– Consider signing up for my mailing list below. You’ll get new post notifications and perhaps even some content that I don’t post on the blog. You can unsubscribe at any time.
If you do either or both of those things, you’d make me happy. Thanks!
